2024-02-10T07:13:45.468 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T07:13:45.469 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --verbose --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204 --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/orig.config.yaml 2024-02-10T07:13:45.499 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T07:13:45.589 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: yweinste@redhat.com first_in_suite: false job_id: '7555204' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6r+X1Nc51VPuwiRcnoeeWcCXT190kaw/sYXARFA9Cow+o1Fp1br3UFSmKFTNSfSrwqVbw8JsKpkseSY9Jz95I= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxmpSynvc11KbCIh4D0t5/jWjiN0YDJnfffjInZO6k8Mqeq1T2+IifrX87mOIZ7LYzOVzRtr122ijg68/P/awQ= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T07:13:45.589 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T07:13:45.589 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T07:13:45.590 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T07:13:45.591 INFO:teuthology.task.internal:Checking packages... 2024-02-10T07:13:45.615 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T07:13:45.615 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T07:13:45.615 INFO:teuthology.packaging:ref: None 2024-02-10T07:13:45.615 INFO:teuthology.packaging:tag: None 2024-02-10T07:13:45.615 INFO:teuthology.packaging:branch: main 2024-02-10T07:13:45.615 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:13:45.615 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T07:13:45.899 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T07:13:45.900 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T07:13:45.907 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T07:13:45.907 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T07:13:45.913 INFO:teuthology.task.internal:Saving configuration 2024-02-10T07:13:45.925 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T07:13:45.972 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:13:45.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:40.582474', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6r+X1Nc51VPuwiRcnoeeWcCXT190kaw/sYXARFA9Cow+o1Fp1br3UFSmKFTNSfSrwqVbw8JsKpkseSY9Jz95I='} 2024-02-10T07:13:46.018 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:40.583937', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxmpSynvc11KbCIh4D0t5/jWjiN0YDJnfffjInZO6k8Mqeq1T2+IifrX87mOIZ7LYzOVzRtr122ijg68/P/awQ='} 2024-02-10T07:13:46.018 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T07:13:46.024 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T07:13:46.024 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-10T07:13:46.024 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T07:13:46.093 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb1d775ef70>, signals=[15]) 2024-02-10T07:13:46.094 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T07:13:46.099 INFO:teuthology.task.internal:Opening connections... 2024-02-10T07:13:46.100 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T07:13:46.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:46.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-02-10T07:13:46.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:13:46.295 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T07:13:46.301 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-02-10T07:13:46.320 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-02-10T07:13:46.320 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-02-10T07:13:46.374 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-02-10T07:13:46.374 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-02-10T07:13:46.374 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T07:13:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T07:13:46.376 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-02-10T07:13:46.398 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-02-10T07:13:46.415 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-02-10T07:13:46.416 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-02-10T07:13:46.476 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-02-10T07:13:46.476 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-02-10T07:13:46.476 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-02-10T07:13:46.476 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-02-10T07:13:46.476 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T07:13:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T07:13:46.478 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-02-10T07:13:46.497 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T07:13:46.506 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T07:13:46.511 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T07:13:46.511 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-02-10T07:13:46.513 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-02-10T07:13:46.535 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T07:13:46.629 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T07:13:46.629 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-02-10T07:13:46.631 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-02-10T07:13:46.720 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T07:13:46.776 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T07:13:46.791 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T07:13:46.791 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T07:13:46.791 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T07:13:46.792 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T07:13:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:13:46.808 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-02-10T07:13:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:13:46.809 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-02-10T07:13:46.865 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-02-10T07:13:46.865 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-02-10T07:13:46.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-02-10T07:13:46.868 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-361.el9.x86_64 2024-02-10T07:13:46.868 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-361.el9.x86_64 2024-02-10T07:13:46.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-02-10T07:13:48.931 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T07:13:49.041 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T07:13:50.390 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-02-10T07:13:50.683 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-02-10T07:13:51.440 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-02-10T07:13:52.464 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-02-10T07:13:54.506 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-02-10T07:13:54.558 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 19 MB 00:02 2024-02-10T07:13:58.407 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 17 MB/s | 5.9 MB 00:00 2024-02-10T07:13:59.175 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 5.9 MB 00:01 2024-02-10T07:13:59.849 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-02-10T07:14:00.752 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2024-02-10T07:14:01.696 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2024-02-10T07:14:06.652 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-02-10T07:14:08.975 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T07:14:08.975 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T07:14:09.070 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-02-10T07:14:09.075 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-02-10T07:14:09.075 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T07:14:09.076 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:14:09.077 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-02-10T07:14:09.077 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-02-10T07:14:09.077 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-02-10T07:14:09.077 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:14:09.078 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 99 M 2024-02-10T07:14:09.078 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 122 M 2024-02-10T07:14:09.078 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-02-10T07:14:10.786 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 3.5 MB/s | 6.0 MB 00:01 2024-02-10T07:14:11.611 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 8.2 MB/s | 21 MB 00:02 2024-02-10T07:14:13.420 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 13 MB/s | 33 MB 00:02 2024-02-10T07:14:13.828 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 8.2 MB/s | 39 MB 00:04 2024-02-10T07:14:13.831 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:14:13.831 INFO:teuthology.orchestra.run.smithi136.stdout:Total 21 MB/s | 99 MB 00:04 2024-02-10T07:14:13.832 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-02-10T07:14:14.224 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-02-10T07:14:14.224 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-02-10T07:14:15.259 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-02-10T07:14:15.260 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-02-10T07:14:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-02-10T07:14:18.409 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:14:18.548 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:14:21.318 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:14:21.502 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:14:26.175 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:14:26.363 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:14:31.097 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:14:43.549 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 478 kB/s | 20 MB 00:43 2024-02-10T07:14:48.512 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-02-10T07:14:50.844 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T07:14:50.844 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T07:14:50.944 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T07:14:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:14:50.952 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-02-10T07:14:50.952 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-02-10T07:14:50.952 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-02-10T07:14:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:14:50.952 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 99 M 2024-02-10T07:14:50.952 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 122 M 2024-02-10T07:14:50.953 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-02-10T07:14:51.355 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-02-10T07:14:51.931 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-02-10T07:14:52.581 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 27 MB/s | 33 MB 00:01 2024-02-10T07:14:53.031 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-02-10T07:14:53.034 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:14:53.034 INFO:teuthology.orchestra.run.smithi064.stdout:Total 48 MB/s | 99 MB 00:02 2024-02-10T07:14:53.035 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-02-10T07:14:53.485 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-02-10T07:14:53.485 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-02-10T07:14:54.556 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-02-10T07:14:54.556 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-02-10T07:14:57.097 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-02-10T07:14:57.776 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:14:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:00.387 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:00.387 INFO:teuthology.orchestra.run.smithi136.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T07:15:00.387 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:15:00.458 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:00.851 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:00.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:15:00.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:00.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:00.973 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:15:01.310 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-02-10T07:15:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T07:15:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T07:15:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T07:15:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:15:01.311 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-02-10T07:15:01.844 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:15:01.968 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:01.968 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:15:01.968 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T07:15:01.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T07:15:01.969 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T07:15:01.969 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T07:15:01.969 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T07:15:01.969 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:15:02.076 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:02.077 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:15:02.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T07:15:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T07:15:02.121 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T07:15:02.122 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-02-10T07:15:02.122 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:15:02.122 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T07:15:02.122 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T07:15:02.123 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-419.el9.x86_64 2024-02-10T07:15:02.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T07:15:02.717 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-02-10T07:15:04.184 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T07:15:04.218 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-02-10T07:15:04.221 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-02-10T07:15:04.236 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.nfaGdBngnW 2024-02-10T07:15:04.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nfaGdBngnW 2024-02-10T07:15:04.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.nfaGdBngnW 2024-02-10T07:15:04.443 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.nfaGdBngnW is 6KB 2024-02-10T07:15:04.464 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.nfaGdBngnW 2024-02-10T07:15:04.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T07:15:04.542 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T07:15:04.543 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T07:15:04.543 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T07:15:04.543 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T07:15:04.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T07:15:04.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-02-10T07:15:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:06.136 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:11.230 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:34.694 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T07:15:34.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:15:34.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:15:42.516 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:42.516 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T07:15:42.516 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:15:42.587 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:43.023 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:43.023 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:15:43.023 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:43.023 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:15:43.453 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-02-10T07:15:43.986 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:15:44.160 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:44.161 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:15:44.161 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T07:15:44.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T07:15:44.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T07:15:44.161 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T07:15:44.161 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T07:15:44.161 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:15:44.273 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:44.274 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:15:44.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-02-10T07:15:44.312 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-02-10T07:15:44.312 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-02-10T07:15:44.312 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-02-10T07:15:44.312 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-02-10T07:15:44.312 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-02-10T07:15:44.312 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T07:15:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T07:15:44.314 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T07:15:44.314 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-02-10T07:15:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:15:44.314 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T07:15:44.315 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T07:15:44.315 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-419.el9.x86_64 2024-02-10T07:15:44.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T07:15:44.957 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-02-10T07:15:46.416 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T07:15:46.437 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-02-10T07:15:46.440 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-02-10T07:15:46.456 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.1iFGscNxi5 2024-02-10T07:15:46.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1iFGscNxi5 2024-02-10T07:15:46.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.1iFGscNxi5 2024-02-10T07:15:46.673 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.1iFGscNxi5 is 6KB 2024-02-10T07:15:46.722 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.1iFGscNxi5 2024-02-10T07:15:46.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T07:15:46.806 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T07:15:46.806 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T07:15:46.806 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T07:15:46.806 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T07:15:46.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T07:15:46.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-02-10T07:15:47.051 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-02-10T07:15:53.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-02-10T07:16:02.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:16:02.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:05.088 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-02-10T07:16:16.961 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T07:16:16.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T07:16:16.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:17.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:16:17.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:35.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-02-10T07:16:35.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-02-10T07:16:44.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T07:16:44.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:47.580 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-02-10T07:16:50.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:16:50.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:50.833 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-02-10T07:16:51.314 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:16:51.315 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T07:16:51.315 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-02-10T07:16:51.328 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-419.el9.x86_64 2024-02-10T07:16:51.328 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T07:16:51.329 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T07:16:51.329 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T07:16:52.329 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T07:16:52.330 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T07:16:52.443 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-02-10T07:16:52.571 DEBUG:teuthology.parallel:result is None 2024-02-10T07:16:59.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T07:16:59.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:31.557 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-02-10T07:17:31.959 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T07:17:31.959 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T07:17:31.959 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-02-10T07:17:31.973 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-419.el9.x86_64 2024-02-10T07:17:31.973 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T07:17:31.973 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T07:17:31.973 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T07:17:32.974 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T07:17:32.975 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T07:17:33.059 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-02-10T07:17:33.098 DEBUG:teuthology.parallel:result is None 2024-02-10T07:17:33.099 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T07:17:33.107 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T07:17:33.107 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T07:17:33.109 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T07:17:33.128 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T07:17:33.137 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T07:17:33.146 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T07:17:33.146 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T07:17:33.165 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T07:17:33.219 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T07:17:33.232 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T07:17:33.233 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T07:17:33.237 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T07:17:33.289 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:17:33.295 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:17:33.304 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:17:33.308 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:17:33.309 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T07:17:33.330 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T07:17:33.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T07:17:33.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T07:17:33.364 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T07:17:33.373 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T07:17:33.373 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T07:17:33.397 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T07:17:33.420 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:17:33.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:17:33.543 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:17:33.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:17:33.661 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:17:33.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T07:17:33.724 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:17:33.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:17:33.823 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:17:33.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:17:33.950 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:17:33.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T07:17:34.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-02-10T07:17:34.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-02-10T07:17:34.077 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:17:34.112 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:17:34.537 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T07:17:34.545 INFO:teuthology.task.internal:Starting timer... 2024-02-10T07:17:34.545 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T07:17:34.558 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T07:17:34.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-02-10T07:17:34.659 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-02-10T07:17:34.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-02-10T07:17:34.722 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-02-10T07:17:34.724 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T07:17:34.724 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-02-10T07:17:34.755 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-02-10T07:17:34.756 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-02-10T07:17:34.813 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-02-10T07:17:34.814 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-02-10T07:17:34.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:17:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:17:34.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:17:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:17:34.898 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T07:17:34.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-02-10T07:17:35.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-02-10T07:17:35.156 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T07:17:35.168 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T07:17:35.185 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T07:17:35.186 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T07:21:48.330 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-02-10T07:21:48.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T07:21:48.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:21:48.423 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-02-10T07:21:48.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T07:21:48.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:21:48.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:21:48.603 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-02-10T07:21:48.684 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-02-10T07:21:48.684 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T07:21:48.695 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T07:21:48.696 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T07:21:48.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:21:48.699 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T07:21:48.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:21:48.727 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T07:21:48.744 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T07:21:48.757 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T07:21:48.774 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T07:21:48.774 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-02-10T07:21:48.787 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-02-10T07:21:48.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-02-10T07:21:48.803 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T07:21:48.816 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-02-10T07:21:48.820 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T07:21:48.833 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T07:21:48.849 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T07:21:48.875 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:21:48.878 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:21:48.878 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-02-10T07:21:48.878 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.879 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.879 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.879 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.903 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:21:48.907 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:21:48.907 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-02-10T07:21:48.907 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.907 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.907 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.907 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:21:48.907 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T07:21:49.016 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T07:21:49.017 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-10T07:21:49.017 INFO:tasks.cephadm:Cluster fsid is 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:21:49.017 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T07:21:49.017 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.64', 'mon.c': '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]', 'mon.b': '172.21.15.136'} 2024-02-10T07:21:49.018 INFO:tasks.cephadm:First mon is mon.a on smithi064 2024-02-10T07:21:49.018 INFO:tasks.cephadm:First mgr is y 2024-02-10T07:21:49.018 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T07:21:49.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-02-10T07:21:49.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-02-10T07:21:49.074 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-10T07:21:49.074 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T07:21:49.320 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 07:21 /home/ubuntu/cephtest/cephadm 2024-02-10T07:21:49.320 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T07:21:49.486 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 07:21 /home/ubuntu/cephtest/cephadm 2024-02-10T07:21:49.486 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T07:21:49.502 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T07:21:49.528 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-10T07:21:49.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T07:21:49.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T07:21:50.346 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T07:21:50.432 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T07:22:09.722 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:22:09.722 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T07:22:09.722 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T07:22:09.722 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-02-10T07:22:09.722 INFO:teuthology.orchestra.run.smithi064.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T07:22:09.723 INFO:teuthology.orchestra.run.smithi064.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T07:22:09.723 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-02-10T07:22:09.723 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-02-10T07:22:11.699 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-02-10T07:22:11.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-02-10T07:22:11.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-02-10T07:22:11.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-02-10T07:22:11.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-02-10T07:22:11.851 INFO:tasks.cephadm:Writing seed config... 2024-02-10T07:22:11.852 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-10T07:22:11.852 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T07:22:11.853 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T07:22:11.854 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:22:11.854 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T07:22:11.879 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0ded8f14-c7e5-11ee-95b8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T07:22:11.880 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T07:22:11.922 DEBUG:teuthology.orchestra.run.smithi064:mgr.y> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T07:22:11.965 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T07:22:11.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.64 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:22:12.168 INFO:teuthology.orchestra.run.smithi064.stderr:-------------------------------------------------------------------------------- 2024-02-10T07:22:12.168 INFO:teuthology.orchestra.run.smithi064.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '0ded8f14-c7e5-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.64', '--skip-admin-label'] 2024-02-10T07:22:12.192 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: 4.9.3-dev 2024-02-10T07:22:12.194 INFO:teuthology.orchestra.run.smithi064.stderr:Verifying podman|docker is present... 2024-02-10T07:22:12.217 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: 4.9.3-dev 2024-02-10T07:22:12.219 INFO:teuthology.orchestra.run.smithi064.stderr:Verifying lvm2 is present... 2024-02-10T07:22:12.219 INFO:teuthology.orchestra.run.smithi064.stderr:Verifying time synchronization is in place... 2024-02-10T07:22:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T07:22:12.232 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: inactive 2024-02-10T07:22:12.238 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: enabled 2024-02-10T07:22:12.244 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: active 2024-02-10T07:22:12.245 INFO:teuthology.orchestra.run.smithi064.stderr:Unit chronyd.service is enabled and running 2024-02-10T07:22:12.245 INFO:teuthology.orchestra.run.smithi064.stderr:Repeating the final host check... 2024-02-10T07:22:12.265 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: 4.9.3-dev 2024-02-10T07:22:12.267 INFO:teuthology.orchestra.run.smithi064.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-10T07:22:12.267 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl is present 2024-02-10T07:22:12.267 INFO:teuthology.orchestra.run.smithi064.stderr:lvcreate is present 2024-02-10T07:22:12.272 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T07:22:12.278 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: inactive 2024-02-10T07:22:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: enabled 2024-02-10T07:22:12.290 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: active 2024-02-10T07:22:12.291 INFO:teuthology.orchestra.run.smithi064.stderr:Unit chronyd.service is enabled and running 2024-02-10T07:22:12.291 INFO:teuthology.orchestra.run.smithi064.stderr:Host looks OK 2024-02-10T07:22:12.291 INFO:teuthology.orchestra.run.smithi064.stderr:Cluster fsid: 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:12.292 INFO:teuthology.orchestra.run.smithi064.stderr:Acquiring lock 140499366084080 on /run/cephadm/0ded8f14-c7e5-11ee-95b8-87774f69a715.lock 2024-02-10T07:22:12.292 INFO:teuthology.orchestra.run.smithi064.stderr:Lock 140499366084080 acquired on /run/cephadm/0ded8f14-c7e5-11ee-95b8-87774f69a715.lock 2024-02-10T07:22:12.292 INFO:teuthology.orchestra.run.smithi064.stderr:Verifying IP 172.21.15.64 port 3300 ... 2024-02-10T07:22:12.292 INFO:teuthology.orchestra.run.smithi064.stderr:Verifying IP 172.21.15.64 port 6789 ... 2024-02-10T07:22:12.292 INFO:teuthology.orchestra.run.smithi064.stderr:Base mon IP is 172.21.15.64, final addrv is [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-02-10T07:22:12.294 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.64 metric 100 2024-02-10T07:22:12.294 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2024-02-10T07:22:12.297 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T07:22:12.297 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T07:22:12.297 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1734sec hoplimit 64 pref medium 2024-02-10T07:22:12.300 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T07:22:12.300 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T07:22:12.300 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T07:22:12.300 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T07:22:12.300 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: inet6 fe80::ec4:7aff:feb6:f8df/64 scope link 2024-02-10T07:22:12.300 INFO:teuthology.orchestra.run.smithi064.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T07:22:12.301 INFO:teuthology.orchestra.run.smithi064.stderr:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-02-10T07:22:12.301 INFO:teuthology.orchestra.run.smithi064.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T07:22:12.301 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T07:22:12.405 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-10T07:22:12.749 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Getting image source signatures 2024-02-10T07:22:12.749 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-10T07:22:12.749 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-10T07:22:12.750 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-10T07:22:12.750 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-10T07:22:12.750 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-10T07:22:12.750 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T07:22:12.918 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T07:22:13.070 INFO:teuthology.orchestra.run.smithi064.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T07:22:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T07:22:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T07:22:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T07:22:14.319 INFO:teuthology.orchestra.run.smithi064.stderr:stat: 167 167 2024-02-10T07:22:14.816 INFO:teuthology.orchestra.run.smithi064.stderr:Creating initial keys... 2024-02-10T07:22:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph-authtool: AQAnJMdl2ClwBBAAaeQrt1AptVAVfwZ/bvIrpQ== 2024-02-10T07:22:15.939 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph-authtool: AQAnJMdle//aMxAACr35nZAux6uFKnB6U0vjjg== 2024-02-10T07:22:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph-authtool: AQAoJMdljZDXLRAAkPZr2rCoSHt0KLPrd6Js5g== 2024-02-10T07:22:17.293 INFO:teuthology.orchestra.run.smithi064.stderr:Creating initial monmap... 2024-02-10T07:22:17.659 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T07:22:17.659 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-10T07:22:17.659 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:17.659 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T07:22:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:monmaptool for a [v2:172.21.15.64:3300,v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T07:22:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:setting min_mon_release = octopus 2024-02-10T07:22:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/monmaptool: set fsid to 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T07:22:18.123 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:22:18.123 INFO:teuthology.orchestra.run.smithi064.stderr:Creating mon... 2024-02-10T07:22:18.987 INFO:teuthology.orchestra.run.smithi064.stderr:create mon.a on 2024-02-10T07:22:19.194 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T07:22:19.342 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715.target. 2024-02-10T07:22:19.342 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715.target. 2024-02-10T07:22:19.620 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Failed to reset failed state of unit ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T07:22:19.627 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715.target.wants/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@.service. 2024-02-10T07:22:20.251 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: disabled 2024-02-10T07:22:20.256 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: inactive 2024-02-10T07:22:20.257 INFO:teuthology.orchestra.run.smithi064.stderr:firewalld.service is not enabled 2024-02-10T07:22:20.257 INFO:teuthology.orchestra.run.smithi064.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T07:22:20.258 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for mon to start... 2024-02-10T07:22:20.258 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for mon... 2024-02-10T07:22:20.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:20 smithi064 systemd[1]: Started Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: cluster: 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: id: 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: services: 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.599246s) 2024-02-10T07:22:20.819 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: data: 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: pgs: 2024-02-10T07:22:20.820 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:21.314 INFO:teuthology.orchestra.run.smithi064.stderr:mon is available 2024-02-10T07:22:21.314 INFO:teuthology.orchestra.run.smithi064.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T07:22:21.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:21 smithi064 ceph-mon[24417]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T07:22:21.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:21 smithi064 ceph-mon[24417]: monmap e1: 1 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0]} 2024-02-10T07:22:21.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:21 smithi064 ceph-mon[24417]: fsmap 2024-02-10T07:22:21.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:21 smithi064 ceph-mon[24417]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T07:22:21.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:21 smithi064 ceph-mon[24417]: mgrmap e1: no daemons active 2024-02-10T07:22:21.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:21 smithi064 ceph-mon[24417]: from='client.? 172.21.15.64:0/604150015' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T07:22:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: [global] 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: fsid = 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: [mgr] 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: [osd] 2024-02-10T07:22:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T07:22:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T07:22:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:Generating new minimal ceph.conf... 2024-02-10T07:22:22.445 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:22 smithi064 ceph-mon[24417]: from='client.? 172.21.15.64:0/3235043695' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T07:22:22.445 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:22 smithi064 ceph-mon[24417]: from='client.? 172.21.15.64:0/3235043695' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T07:22:23.301 INFO:teuthology.orchestra.run.smithi064.stderr:Restarting the monitor... 2024-02-10T07:22:23.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 ceph-mon[24417]: from='client.? 172.21.15.64:0/567938868' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:22:23.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 systemd[1]: Stopping Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:22:23.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 bash[24635]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T07:22:23.777 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a[24413]: 2024-02-10T07:22:23.460+0000 7fbada56d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:22:23.777 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a[24413]: 2024-02-10T07:22:23.460+0000 7fbada56d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T07:22:23.777 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 podman[24646]: 2024-02-10 07:22:23.531568073 +0000 UTC m=+0.173257323 container died 7af259416a31d0fba21e3b33065ce1314c32f001c45b652328c81d337d81717e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, name=centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, release=754, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8) 2024-02-10T07:22:24.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 podman[24646]: 2024-02-10 07:22:23.876503082 +0000 UTC m=+0.518192340 container cleanup 7af259416a31d0fba21e3b33065ce1314c32f001c45b652328c81d337d81717e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_CLEAN=True, com.redhat.component=centos-stream-container, release=754) 2024-02-10T07:22:24.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 bash[24646]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a 2024-02-10T07:22:24.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:23 smithi064 bash[24672]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T07:22:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 podman[24660]: 2024-02-10 07:22:24.110528153 +0000 UTC m=+0.582141055 container remove 7af259416a31d0fba21e3b33065ce1314c32f001c45b652328c81d337d81717e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, name=centos-stream, version=8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, release=754, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, ceph=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-10T07:22:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T07:22:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 systemd[1]: Stopped Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:22:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 systemd[1]: Starting Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:22:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 podman[24741]: 2024-02-10 07:22:24.344620354 +0000 UTC m=+0.102687906 container create 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, version=8, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, release=754, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T07:22:24.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 podman[24741]: 2024-02-10 07:22:24.260955499 +0000 UTC m=+0.019023049 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-10T07:22:24.633 INFO:teuthology.orchestra.run.smithi064.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-10T07:22:24.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 podman[24741]: 2024-02-10 07:22:24.546863034 +0000 UTC m=+0.304930577 container init 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, release=754, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, distribution-scope=public, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T07:22:24.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 podman[24741]: 2024-02-10 07:22:24.549709642 +0000 UTC m=+0.307777185 container start 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, distribution-scope=public, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T07:22:24.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:22:24.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: pidfile_write: ignore empty --pid-file 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: load: jerasure load: lrc 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: RocksDB version: 6.15.5 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Compile date Apr 18 2022 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: DB SUMMARY 2024-02-10T07:22:24.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: DB Session ID: X5AA7SO8BLGUCB11WIUM 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82627 ; 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.env: 0x563ccaac6860 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.fs: Posix File System 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.info_log: 0x563ccc19fdc0 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:22:24.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.statistics: (nil) 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.use_fsync: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.db_log_dir: 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T07:22:24.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.write_buffer_manager: 0x563ccc2902a0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:22:24.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.unordered_write: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.row_cache: None 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.wal_filter: None 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.preserve_deletes: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:22:24.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:22:24.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:22:24.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:22:24.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:22:24.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_open_files: -1 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:22:24.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Compression algorithms supported: 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kZSTD supported: 0 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:22:24.748 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.merge_operator: 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_filter: None 2024-02-10T07:22:24.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563ccc16bd00) 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-02-10T07:22:24.750 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: hash_index_allow_collision: 1 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 1 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x563ccc1d6170 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-02-10T07:22:24.751 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 4 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression: NoCompression 2024-02-10T07:22:24.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.num_levels: 7 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:22:24.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:22:24.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:22:24.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:22:24.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.ttl: 2592000 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:22:24.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549744574797, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549744575486, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79177, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77440, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71749, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707549744, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cacce143-0300-42d2-8488-c30d2e2fe238", "db_session_id": "X5AA7SO8BLGUCB11WIUM"}} 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549744575811, "job": 1, "event": "recovery_finished"} 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563ccc1bca80 2024-02-10T07:22:24.758 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: DB pointer 0x563ccc1cc000 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:22:24.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:22:24.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 26.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 26.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.761 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 25.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:22:24.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???) e1 preinit fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).mds e1 new map 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).mds e1 print_map 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T07:22:24.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 bash[24741]: 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 2024-02-10T07:22:24.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 systemd[1]: Started Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:22:24.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T07:22:24.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: monmap e1: 1 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0]} 2024-02-10T07:22:24.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: fsmap 2024-02-10T07:22:24.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T07:22:24.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:24 smithi064 ceph-mon[24760]: mgrmap e1: no daemons active 2024-02-10T07:22:25.648 INFO:teuthology.orchestra.run.smithi064.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T07:22:25.648 INFO:teuthology.orchestra.run.smithi064.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:22:25.648 INFO:teuthology.orchestra.run.smithi064.stderr:Creating mgr... 2024-02-10T07:22:25.648 INFO:teuthology.orchestra.run.smithi064.stderr:Verifying port 9283 ... 2024-02-10T07:22:25.816 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Failed to reset failed state of unit ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T07:22:25.822 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715.target.wants/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@.service. 2024-02-10T07:22:26.175 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:25 smithi064 systemd[1]: Starting Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:22:26.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:26 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3112599396' entity='client.admin' 2024-02-10T07:22:26.449 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 podman[24979]: 2024-02-10 07:22:26.18782771 +0000 UTC m=+0.107302678 container create 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, ceph=True, vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, architecture=x86_64, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, vcs-type=git, io.buildah.version=1.19.8) 2024-02-10T07:22:26.449 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 podman[24979]: 2024-02-10 07:22:26.098527769 +0000 UTC m=+0.018002688 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-10T07:22:26.449 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 podman[24979]: 2024-02-10 07:22:26.381634198 +0000 UTC m=+0.301109107 container init 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , release=754, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8) 2024-02-10T07:22:26.449 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 podman[24979]: 2024-02-10 07:22:26.384130321 +0000 UTC m=+0.303605239 container start 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, release=754, com.redhat.component=centos-stream-container, distribution-scope=public, RELEASE=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64) 2024-02-10T07:22:26.449 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 bash[24979]: 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 2024-02-10T07:22:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: disabled 2024-02-10T07:22:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: inactive 2024-02-10T07:22:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:firewalld.service is not enabled 2024-02-10T07:22:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T07:22:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: disabled 2024-02-10T07:22:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: inactive 2024-02-10T07:22:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:firewalld.service is not enabled 2024-02-10T07:22:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T07:22:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for mgr to start... 2024-02-10T07:22:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for mgr... 2024-02-10T07:22:26.712 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 systemd[1]: Started Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:22:26.978 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:26.885+0000 7fa23a952000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:22:26.978 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:26 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:26.978+0000 7fa23a952000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "fsid": "0ded8f14-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "health": { 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 0 2024-02-10T07:22:26.987 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "a" 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T07:22:26.988 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T07:22:26.989 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "available": false, 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "iostat", 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "nfs", 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "restful" 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "services": {} 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T07:22:26.992 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:26.992 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "modified": "2024-02-10T07:22:20.219616+0000", 2024-02-10T07:22:26.992 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "services": {} 2024-02-10T07:22:26.992 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:26.992 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T07:22:26.992 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:22:27.293 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:27 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2083272110' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:22:27.524 INFO:teuthology.orchestra.run.smithi064.stderr:mgr not available, waiting (1/15)... 2024-02-10T07:22:27.927 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:27.602+0000 7fa23a952000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:22:27.927 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:27.692+0000 7fa23a952000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:22:27.927 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:27.777+0000 7fa23a952000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:22:28.204 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:27.927+0000 7fa23a952000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:22:28.204 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:28.009+0000 7fa23a952000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:22:28.205 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:28.119+0000 7fa23a952000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:22:28.205 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:28.204+0000 7fa23a952000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:22:28.456 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:28.369+0000 7fa23a952000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:22:28.456 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:28.456+0000 7fa23a952000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:22:29.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:29 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:29.524+0000 7fa23a952000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:22:29.815 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:29 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:29.661+0000 7fa23a952000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:22:30.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:29 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:29.854+0000 7fa23a952000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:22:30.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:29 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:29.942+0000 7fa23a952000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:22:30.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:30.082+0000 7fa23a952000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:22:30.714 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:30.403+0000 7fa23a952000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:22:30.714 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:30.628+0000 7fa23a952000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:22:30.714 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:30.714+0000 7fa23a952000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:22:31.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:30.889+0000 7fa23a952000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:22:32.153 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:31 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:31.831+0000 7fa23a952000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:22:32.153 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:32.135+0000 7fa23a952000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:22:32.451 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:32.231+0000 7fa23a952000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:22:32.451 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:32.384+0000 7fa23a952000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:22:32.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: Activating manager daemon y 2024-02-10T07:22:32.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: mgrmap e2: y(active, starting, since 0.00286976s) 2024-02-10T07:22:32.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:22:32.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:22:32.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:22:32.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: Manager daemon y is now available 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' 2024-02-10T07:22:32.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:32 smithi064 ceph-mon[24760]: from='mgr.14100 172.21.15.64:0/3694342746' entity='mgr.y' 2024-02-10T07:22:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:22:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "fsid": "0ded8f14-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:22:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "health": { 2024-02-10T07:22:33.455 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 0 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "a" 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-10T07:22:33.456 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.457 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.458 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "available": false, 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "iostat", 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "nfs", 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "restful" 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "services": {} 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T07:22:33.459 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:33.460 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "modified": "2024-02-10T07:22:20.219616+0000", 2024-02-10T07:22:33.460 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "services": {} 2024-02-10T07:22:33.460 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:33.460 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T07:22:33.460 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:22:33.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:33 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1928111494' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:22:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:33 smithi064 ceph-mon[24760]: mgrmap e3: y(active, since 1.00658s) 2024-02-10T07:22:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:35 smithi064 ceph-mon[24760]: mgrmap e4: y(active, since 2s) 2024-02-10T07:22:39.067 INFO:teuthology.orchestra.run.smithi064.stderr:mgr not available, waiting (2/15)... 2024-02-10T07:22:41.728 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:41.728 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "fsid": "0ded8f14-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "health": { 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 0 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "a" 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "quorum_age": 17, 2024-02-10T07:22:41.729 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T07:22:41.730 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "available": true, 2024-02-10T07:22:41.731 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "iostat", 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "nfs", 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "restful" 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ], 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "services": {} 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "modified": "2024-02-10T07:22:20.219616+0000", 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "services": {} 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: }, 2024-02-10T07:22:41.732 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T07:22:41.733 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:22:41.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:41 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4051458312' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:22:42.262 INFO:teuthology.orchestra.run.smithi064.stderr:mgr is available 2024-02-10T07:22:42.876 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: [global] 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: fsid = 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: [mgr] 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 2024-02-10T07:22:42.877 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: [osd] 2024-02-10T07:22:42.878 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T07:22:42.878 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T07:22:43.207 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:42 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3564892302' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T07:22:43.207 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:42 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3564892302' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T07:22:43.486 INFO:teuthology.orchestra.run.smithi064.stderr:Enabling cephadm module... 2024-02-10T07:22:44.446 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:44 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1797929799' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T07:22:44.447 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setuser ceph since I am not root 2024-02-10T07:22:44.447 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setgroup ceph since I am not root 2024-02-10T07:22:44.772 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:44.668+0000 7f3a03c9c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:22:45.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:44.773+0000 7f3a03c9c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:22:45.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1797929799' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T07:22:45.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-mon[24760]: mgrmap e5: y(active, since 11s) 2024-02-10T07:22:45.329 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:22:45.329 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T07:22:45.329 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "available": true, 2024-02-10T07:22:45.329 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-10T07:22:45.329 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T07:22:45.329 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:22:45.646 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:45.396+0000 7f3a03c9c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:22:45.646 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:45.487+0000 7f3a03c9c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:22:45.646 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:45.573+0000 7f3a03c9c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:22:45.905 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:45.724+0000 7f3a03c9c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:22:45.905 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:45.807+0000 7f3a03c9c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:22:45.907 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for the mgr to restart... 2024-02-10T07:22:45.907 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for mgr epoch 5... 2024-02-10T07:22:46.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:46 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1523294172' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T07:22:46.181 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:45.920+0000 7f3a03c9c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:22:46.181 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:46 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:46.012+0000 7f3a03c9c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:22:46.182 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:46 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:46.180+0000 7f3a03c9c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:22:46.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:46 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:46.269+0000 7f3a03c9c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:22:47.660 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:47.332+0000 7f3a03c9c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:22:47.660 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:47.465+0000 7f3a03c9c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:22:47.660 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:47.659+0000 7f3a03c9c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:22:47.941 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:47.748+0000 7f3a03c9c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:22:47.941 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:47.888+0000 7f3a03c9c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:22:48.211 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:48 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:48.210+0000 7f3a03c9c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:22:48.526 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:48 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:48.439+0000 7f3a03c9c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:22:48.526 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:48 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:48.525+0000 7f3a03c9c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:22:48.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:48 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:48.700+0000 7f3a03c9c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:22:49.949 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:49 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:49.643+0000 7f3a03c9c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:22:49.949 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:49 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:49.948+0000 7f3a03c9c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:22:50.201 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:50 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:50.045+0000 7f3a03c9c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:22:50.201 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:50 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:22:50.200+0000 7f3a03c9c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:22:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:50 smithi064 ceph-mon[24760]: Active manager daemon y restarted 2024-02-10T07:22:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:50 smithi064 ceph-mon[24760]: Activating manager daemon y 2024-02-10T07:22:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:50 smithi064 ceph-mon[24760]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T07:22:51.599 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:22:51.599 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-10T07:22:51.599 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T07:22:51.599 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: mgrmap e6: y(active, starting, since 0.373263s) 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: Manager daemon y is now available 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:22:51.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:22:51.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:22:51.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:22:51.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:51 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:22:52.163 INFO:teuthology.orchestra.run.smithi064.stderr:mgr epoch 5 is available 2024-02-10T07:22:52.163 INFO:teuthology.orchestra.run.smithi064.stderr:Setting orchestrator backend to cephadm... 2024-02-10T07:22:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:52 smithi064 ceph-mon[24760]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T07:22:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:52 smithi064 ceph-mon[24760]: mgrmap e7: y(active, since 1.37829s) 2024-02-10T07:22:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:52 smithi064 ceph-mon[24760]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T07:22:53.140 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:22:53] ENGINE Bus STARTING 2024-02-10T07:22:53.493 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:22:53] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:22:53.493 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:22:53] ENGINE Bus STARTED 2024-02-10T07:22:53.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: mgrmap e8: y(active, since 2s) 2024-02-10T07:22:53.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:53.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:22:53.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: [10/Feb/2024:07:22:53] ENGINE Bus STARTING 2024-02-10T07:22:53.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: [10/Feb/2024:07:22:53] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:22:53.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: [10/Feb/2024:07:22:53] ENGINE Bus STARTED 2024-02-10T07:22:53.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:53.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:53 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:22:53.972 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: value unchanged 2024-02-10T07:22:54.492 INFO:teuthology.orchestra.run.smithi064.stderr:Generating ssh key... 2024-02-10T07:22:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:54 smithi064 ceph-mon[24760]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: Generating public/private rsa key pair. 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: Your identification has been saved in /tmp/tmp9g9axy5l/key. 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: Your public key has been saved in /tmp/tmp9g9axy5l/key.pub. 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: The key fingerprint is: 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: SHA256:sfAhoocq1cvHGK+9khSzHvyTWJaMclLaWTE1DZj8k4I ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: The key's randomart image is: 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: +---[RSA 3072]----+ 2024-02-10T07:22:55.438 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | . ++o | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | * .. | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | ..o+o. | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | +E.o+++ | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | +=+O oS. | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | o+oX*= | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: |o *+B+. | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: |. =++ | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: | ..oo | 2024-02-10T07:22:55.439 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:22:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: +----[SHA256]-----+ 2024-02-10T07:22:56.267 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKrgigV4/E33LwGhlevL+7IHYCqDODVBXrfyv7rqP/+OFfWBK1elYNKd/ZUYNyhvVBWeHCr4uc8CgJKqn/yy/Ip6qTuS3+KMnViVshp/CDCTzxcobmEG94D350hfafvHZ3CR/MEDaa8x33E/xXHb1RbmwDht1Mz4A4tMczYL/cfxQhBpxvibVNLZcMvDS/lNzbhKV/l0MaZ9EugVIeZ6eDvWSfgpznJf9Cv2fLCxSZQIWyfV7fYKypzWsRh+v7wcAViU1ohRqtKW2HoABonAPoue7jdLC6eqgUBKyo6CDpGis2bzCWVbI9lJRhN1z5po2Ks9cX4VSdrVx2z8S7s/vj2q+mMp3Et5edQ9V9P1EiIffyWxazm47d4WxLeKrkZRSMK1Hl81lJb6OmRtCL8GvKemUZiNNeNjKEzR70AdURaN9fmkemPWtDT9wCRnEwFzeyXDIapi8QkW5ffiu8mYcQJvddFEf7xDU9XffO9go/SaYG48xB+otWjhSr84046V8= ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:22:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:56 smithi064 ceph-mon[24760]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:22:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:56 smithi064 ceph-mon[24760]: Generating ssh key... 2024-02-10T07:22:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:56 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:56 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:56.847 INFO:teuthology.orchestra.run.smithi064.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:22:56.847 INFO:teuthology.orchestra.run.smithi064.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T07:22:56.847 INFO:teuthology.orchestra.run.smithi064.stderr:Adding host smithi064... 2024-02-10T07:22:57.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:57 smithi064 ceph-mon[24760]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:22:58.433 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: Added host 'smithi064' with addr '172.21.15.64' 2024-02-10T07:22:58.445 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:58 smithi064 ceph-mon[24760]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:22:58.445 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:58 smithi064 ceph-mon[24760]: Deploying cephadm binary to smithi064 2024-02-10T07:22:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:Deploying unmanaged mon service... 2024-02-10T07:22:59.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:59 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:22:59.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:59 smithi064 ceph-mon[24760]: Added host smithi064 2024-02-10T07:22:59.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:22:59 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:22:59.638 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T07:23:00.413 INFO:teuthology.orchestra.run.smithi064.stderr:Deploying unmanaged mgr service... 2024-02-10T07:23:00.980 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:00 smithi064 ceph-mon[24760]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:00.980 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:00 smithi064 ceph-mon[24760]: Saving service mon spec with placement count:5 2024-02-10T07:23:00.980 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:00 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:23:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T07:23:02.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:02 smithi064 ceph-mon[24760]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:02.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:02 smithi064 ceph-mon[24760]: Saving service mgr spec with placement count:2 2024-02-10T07:23:02.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:02 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:23:02.280 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:02 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:23:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:03 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:23:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:03 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1644297841' entity='client.admin' 2024-02-10T07:23:04.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:04 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:23:05.283 INFO:teuthology.orchestra.run.smithi064.stderr:Enabling the dashboard module... 2024-02-10T07:23:05.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:05 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1371024799' entity='client.admin' 2024-02-10T07:23:05.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:05 smithi064 ceph-mon[24760]: from='mgr.14118 172.21.15.64:0/1518186208' entity='mgr.y' 2024-02-10T07:23:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:06 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1614307370' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T07:23:07.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setuser ceph since I am not root 2024-02-10T07:23:07.065 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setgroup ceph since I am not root 2024-02-10T07:23:07.362 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:07.270+0000 7f3714d68000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:23:07.362 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:07.361+0000 7f3714d68000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:23:07.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:07 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1614307370' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T07:23:07.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:07 smithi064 ceph-mon[24760]: mgrmap e9: y(active, since 16s) 2024-02-10T07:23:08.084 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:23:08.084 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-10T07:23:08.084 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "available": true, 2024-02-10T07:23:08.084 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-10T07:23:08.084 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T07:23:08.084 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:23:08.311 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:07.978+0000 7f3714d68000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:23:08.312 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.068+0000 7f3714d68000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:23:08.312 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.160+0000 7f3714d68000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:23:08.312 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.310+0000 7f3714d68000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:23:08.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.393+0000 7f3714d68000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:23:08.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.503+0000 7f3714d68000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:23:08.626 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for the mgr to restart... 2024-02-10T07:23:08.626 INFO:teuthology.orchestra.run.smithi064.stderr:Waiting for mgr epoch 9... 2024-02-10T07:23:08.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2591450405' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T07:23:08.850 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.588+0000 7f3714d68000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:23:08.850 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.759+0000 7f3714d68000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:23:08.850 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:08.849+0000 7f3714d68000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:23:10.215 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:09 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:09.889+0000 7f3714d68000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:23:10.215 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:10.021+0000 7f3714d68000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:23:10.215 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:10.213+0000 7f3714d68000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:23:10.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:10.302+0000 7f3714d68000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:23:10.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:10.441+0000 7f3714d68000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:23:11.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:10.760+0000 7f3714d68000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:23:11.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:10.986+0000 7f3714d68000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:23:11.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:11.072+0000 7f3714d68000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:23:11.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:11.246+0000 7f3714d68000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:23:12.484 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:12.186+0000 7f3714d68000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:23:12.760 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:12.505+0000 7f3714d68000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:23:12.760 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:12.604+0000 7f3714d68000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:23:12.760 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:23:12.760+0000 7f3714d68000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:23:13.837 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: { 2024-02-10T07:23:13.837 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-10T07:23:13.837 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T07:23:13.838 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: } 2024-02-10T07:23:13.863 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:23:13] ENGINE Bus STARTING 2024-02-10T07:23:13.863 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:23:13] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:23:13.863 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:23:13] ENGINE Bus STARTED 2024-02-10T07:23:13.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: Active manager daemon y restarted 2024-02-10T07:23:13.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: Activating manager daemon y 2024-02-10T07:23:13.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: mgrmap e10: y(active, starting, since 0.0555055s) 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: Manager daemon y is now available 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:23:13.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:14.736 INFO:teuthology.orchestra.run.smithi064.stderr:mgr epoch 9 is available 2024-02-10T07:23:14.736 INFO:teuthology.orchestra.run.smithi064.stderr:Generating a dashboard self-signed certificate... 2024-02-10T07:23:15.004 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:14 smithi064 ceph-mon[24760]: [10/Feb/2024:07:23:13] ENGINE Bus STARTING 2024-02-10T07:23:15.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:14 smithi064 ceph-mon[24760]: [10/Feb/2024:07:23:13] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:23:15.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:14 smithi064 ceph-mon[24760]: [10/Feb/2024:07:23:13] ENGINE Bus STARTED 2024-02-10T07:23:15.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:14 smithi064 ceph-mon[24760]: mgrmap e11: y(active, since 1.05745s) 2024-02-10T07:23:15.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:15.468 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T07:23:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:15 smithi064 ceph-mon[24760]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T07:23:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:15 smithi064 ceph-mon[24760]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T07:23:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:15 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:15 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:15 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:15 smithi064 ceph-mon[24760]: mgrmap e12: y(active, since 3s) 2024-02-10T07:23:16.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:16 smithi064 ceph-mon[24760]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:16.866 INFO:teuthology.orchestra.run.smithi064.stderr:Creating initial admin user... 2024-02-10T07:23:17.772 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$9WpHFFOfLSusUuiq6qdyEuGFSAyXMae9rpTPxk6ba9HM5SrwzeuHC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707549797, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T07:23:18.425 INFO:teuthology.orchestra.run.smithi064.stderr:Fetching dashboard port number... 2024-02-10T07:23:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:18 smithi064 ceph-mon[24760]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: 8443 2024-02-10T07:23:19.739 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: disabled 2024-02-10T07:23:19.746 INFO:teuthology.orchestra.run.smithi064.stderr:systemctl: inactive 2024-02-10T07:23:19.747 INFO:teuthology.orchestra.run.smithi064.stderr:firewalld.service is not enabled 2024-02-10T07:23:19.747 INFO:teuthology.orchestra.run.smithi064.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T07:23:19.749 INFO:teuthology.orchestra.run.smithi064.stderr:Ceph Dashboard is now available at: 2024-02-10T07:23:19.749 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:19.750 INFO:teuthology.orchestra.run.smithi064.stderr: URL: https://smithi064.front.sepia.ceph.com:8443/ 2024-02-10T07:23:19.750 INFO:teuthology.orchestra.run.smithi064.stderr: User: admin 2024-02-10T07:23:19.750 INFO:teuthology.orchestra.run.smithi064.stderr: Password: hgshbnmvac 2024-02-10T07:23:19.750 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:19.750 INFO:teuthology.orchestra.run.smithi064.stderr:Enabling autotune for osd_memory_target 2024-02-10T07:23:19.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:19 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/768961037' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T07:23:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:20 smithi064 ceph-mon[24760]: mgrmap e13: y(active, since 7s) 2024-02-10T07:23:21.849 INFO:teuthology.orchestra.run.smithi064.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-10T07:23:22.352 INFO:teuthology.orchestra.run.smithi064.stderr:You can access the Ceph CLI with: 2024-02-10T07:23:22.352 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:22.352 INFO:teuthology.orchestra.run.smithi064.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: ceph telemetry on 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr:For more information see: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:23:22.353 INFO:teuthology.orchestra.run.smithi064.stderr:Bootstrap complete. 2024-02-10T07:23:22.389 INFO:tasks.cephadm:Fetching config... 2024-02-10T07:23:22.389 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:23:22.389 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T07:23:22.405 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T07:23:22.405 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:23:22.405 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T07:23:22.460 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T07:23:22.460 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:23:22.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T07:23:22.528 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T07:23:22.528 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:23:22.528 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T07:23:22.582 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T07:23:22.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKrgigV4/E33LwGhlevL+7IHYCqDODVBXrfyv7rqP/+OFfWBK1elYNKd/ZUYNyhvVBWeHCr4uc8CgJKqn/yy/Ip6qTuS3+KMnViVshp/CDCTzxcobmEG94D350hfafvHZ3CR/MEDaa8x33E/xXHb1RbmwDht1Mz4A4tMczYL/cfxQhBpxvibVNLZcMvDS/lNzbhKV/l0MaZ9EugVIeZ6eDvWSfgpznJf9Cv2fLCxSZQIWyfV7fYKypzWsRh+v7wcAViU1ohRqtKW2HoABonAPoue7jdLC6eqgUBKyo6CDpGis2bzCWVbI9lJRhN1z5po2Ks9cX4VSdrVx2z8S7s/vj2q+mMp3Et5edQ9V9P1EiIffyWxazm47d4WxLeKrkZRSMK1Hl81lJb6OmRtCL8GvKemUZiNNeNjKEzR70AdURaN9fmkemPWtDT9wCRnEwFzeyXDIapi8QkW5ffiu8mYcQJvddFEf7xDU9XffO9go/SaYG48xB+otWjhSr84046V8= ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T07:23:22.661 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKrgigV4/E33LwGhlevL+7IHYCqDODVBXrfyv7rqP/+OFfWBK1elYNKd/ZUYNyhvVBWeHCr4uc8CgJKqn/yy/Ip6qTuS3+KMnViVshp/CDCTzxcobmEG94D350hfafvHZ3CR/MEDaa8x33E/xXHb1RbmwDht1Mz4A4tMczYL/cfxQhBpxvibVNLZcMvDS/lNzbhKV/l0MaZ9EugVIeZ6eDvWSfgpznJf9Cv2fLCxSZQIWyfV7fYKypzWsRh+v7wcAViU1ohRqtKW2HoABonAPoue7jdLC6eqgUBKyo6CDpGis2bzCWVbI9lJRhN1z5po2Ks9cX4VSdrVx2z8S7s/vj2q+mMp3Et5edQ9V9P1EiIffyWxazm47d4WxLeKrkZRSMK1Hl81lJb6OmRtCL8GvKemUZiNNeNjKEzR70AdURaN9fmkemPWtDT9wCRnEwFzeyXDIapi8QkW5ffiu8mYcQJvddFEf7xDU9XffO9go/SaYG48xB+otWjhSr84046V8= ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:23:22.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKrgigV4/E33LwGhlevL+7IHYCqDODVBXrfyv7rqP/+OFfWBK1elYNKd/ZUYNyhvVBWeHCr4uc8CgJKqn/yy/Ip6qTuS3+KMnViVshp/CDCTzxcobmEG94D350hfafvHZ3CR/MEDaa8x33E/xXHb1RbmwDht1Mz4A4tMczYL/cfxQhBpxvibVNLZcMvDS/lNzbhKV/l0MaZ9EugVIeZ6eDvWSfgpznJf9Cv2fLCxSZQIWyfV7fYKypzWsRh+v7wcAViU1ohRqtKW2HoABonAPoue7jdLC6eqgUBKyo6CDpGis2bzCWVbI9lJRhN1z5po2Ks9cX4VSdrVx2z8S7s/vj2q+mMp3Et5edQ9V9P1EiIffyWxazm47d4WxLeKrkZRSMK1Hl81lJb6OmRtCL8GvKemUZiNNeNjKEzR70AdURaN9fmkemPWtDT9wCRnEwFzeyXDIapi8QkW5ffiu8mYcQJvddFEf7xDU9XffO9go/SaYG48xB+otWjhSr84046V8= ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T07:23:22.723 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKrgigV4/E33LwGhlevL+7IHYCqDODVBXrfyv7rqP/+OFfWBK1elYNKd/ZUYNyhvVBWeHCr4uc8CgJKqn/yy/Ip6qTuS3+KMnViVshp/CDCTzxcobmEG94D350hfafvHZ3CR/MEDaa8x33E/xXHb1RbmwDht1Mz4A4tMczYL/cfxQhBpxvibVNLZcMvDS/lNzbhKV/l0MaZ9EugVIeZ6eDvWSfgpznJf9Cv2fLCxSZQIWyfV7fYKypzWsRh+v7wcAViU1ohRqtKW2HoABonAPoue7jdLC6eqgUBKyo6CDpGis2bzCWVbI9lJRhN1z5po2Ks9cX4VSdrVx2z8S7s/vj2q+mMp3Et5edQ9V9P1EiIffyWxazm47d4WxLeKrkZRSMK1Hl81lJb6OmRtCL8GvKemUZiNNeNjKEzR70AdURaN9fmkemPWtDT9wCRnEwFzeyXDIapi8QkW5ffiu8mYcQJvddFEf7xDU9XffO9go/SaYG48xB+otWjhSr84046V8= ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:23:22.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T07:23:23.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:22 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1616879704' entity='client.admin' 2024-02-10T07:23:26.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:26 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2426528570' entity='client.admin' 2024-02-10T07:23:26.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:26.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:23:26.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:26.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:26.803 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:26.825 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T07:23:26.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T07:23:29.489 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2024-02-10T07:23:29.489 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:23:29.490 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2024-02-10T07:23:29.508 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:23:29.508 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:23:29.565 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2024-02-10T07:23:29.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch host add smithi136 2024-02-10T07:23:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:30.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:31.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:30 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:23:32.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:31 smithi064 ceph-mon[24760]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:32.672 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi136' with addr '172.21.15.136' 2024-02-10T07:23:33.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T07:23:33.772 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: Deploying cephadm binary to smithi136 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: Added host smithi136 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:33.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:34.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:34 smithi064 ceph-mon[24760]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:34.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:34 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:23:35.052 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2024-02-10T07:23:35.555 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T07:23:35.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T07:23:36.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:36 smithi064 ceph-mon[24760]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:36.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:36 smithi064 ceph-mon[24760]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:23:38.166 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-02-10T07:23:38.453 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:38 smithi064 ceph-mon[24760]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:38.453 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:38 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:38.453 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:38 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:38.453 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:38 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:23:38.453 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:38 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:38.453 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:38 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3258403219' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T07:23:38.697 INFO:tasks.cephadm:Adding mon.a on smithi064 2024-02-10T07:23:38.697 INFO:tasks.cephadm:Adding mon.c on smithi064 2024-02-10T07:23:38.697 INFO:tasks.cephadm:Adding mon.b on smithi136 2024-02-10T07:23:38.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b' 2024-02-10T07:23:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:39 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3258403219' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T07:23:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:39 smithi064 ceph-mon[24760]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:23:40.820 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mon update... 2024-02-10T07:23:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:40 smithi064 ceph-mon[24760]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:41.532 DEBUG:teuthology.orchestra.run.smithi064:mon.c> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T07:23:41.534 DEBUG:teuthology.orchestra.run.smithi136:mon.b> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T07:23:41.536 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:23:41.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:23:41.945 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:41.945 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b;count:3 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:23:41.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:41.947 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:41 smithi064 ceph-mon[24760]: Deploying daemon mon.c on smithi064 2024-02-10T07:23:43.451 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:23:43.452 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:22:17.590807Z","created":"2024-02-10T07:22:17.590807Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T07:23:43.453 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-02-10T07:23:44.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:43 smithi064 ceph-mon[24760]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:44.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:43 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/489364838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:44.475 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 podman[29458]: 2024-02-10 07:23:44.294798943 +0000 UTC m=+0.018680734 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-10T07:23:44.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 podman[29458]: 2024-02-10 07:23:44.475332252 +0000 UTC m=+0.199214077 container create ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, io.openshift.expose-services=, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T07:23:44.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 podman[29458]: 2024-02-10 07:23:44.63615708 +0000 UTC m=+0.360038872 container init ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , release=754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.buildah.version=1.19.8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD) 2024-02-10T07:23:44.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 podman[29458]: 2024-02-10 07:23:44.639371986 +0000 UTC m=+0.363253769 container start ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, GIT_CLEAN=True, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-type=git, version=8, io.k8s.display-name=CentOS Stream 8, release=754, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: pidfile_write: ignore empty --pid-file 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: load: jerasure load: lrc 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: RocksDB version: 6.15.5 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Compile date Apr 18 2022 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: DB SUMMARY 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: DB Session ID: O7HNB3INHTZPD55O0LMD 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-10T07:23:44.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.env: 0x5612730c6860 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.fs: Posix File System 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.info_log: 0x561274de5dc0 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.statistics: (nil) 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.use_fsync: 0 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:23:44.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:23:44.736 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.db_log_dir: 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.write_buffer_manager: 0x561274ed62d0 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:23:44.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.unordered_write: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.row_cache: None 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.wal_filter: None 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.preserve_deletes: 0 2024-02-10T07:23:44.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:23:44.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_open_files: -1 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:23:44.740 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Compression algorithms supported: 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kZSTD supported: 0 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T07:23:44.741 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.merge_operator: 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_filter: None 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561274db1d00) 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_type: 0 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_index_type: 0 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_shortening: 1 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:23:44.742 INFO:journalctl@ceph.mon.c.smithi064.stdout: hash_index_allow_collision: 1 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: checksum: 1 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: no_block_cache: 0 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache: 0x561274e1c170 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_options: 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: capacity : 536870912 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_shard_bits : 4 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: strict_capacity_limit : 0 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_compressed: (nil) 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: persistent_cache: (nil) 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size: 4096 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size_deviation: 10 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_restart_interval: 16 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_block_restart_interval: 1 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: metadata_block_size: 4096 2024-02-10T07:23:44.743 INFO:journalctl@ceph.mon.c.smithi064.stdout: partition_filters: 0 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: use_delta_encoding: 1 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: whole_key_filtering: 1 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: verify_compression: 0 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: format_version: 4 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_index_compression: 1 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_align: 0 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression: NoCompression 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:23:44.744 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.num_levels: 7 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:23:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:23:44.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:23:44.747 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:23:44.748 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.ttl: 2592000 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549824665121, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T07:23:44.749 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549824665429, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707549824, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8ed8990b-d744-449f-96ba-44fc8f21e071", "db_session_id": "O7HNB3INHTZPD55O0LMD"}} 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549824665730, "job": 1, "event": "recovery_finished"} 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x561274e02a80 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: DB pointer 0x561274e12000 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** DB Stats ** 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T07:23:44.750 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:44.751 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T07:23:44.752 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.753 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: using public_addrv [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: starting mon.c rank -1 at public addrs [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] at bind addrs [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(???) e0 preinit fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).mds e1 new map 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout: e1 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout: legacy client fscid: -1 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout: No filesystems configured 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T07:23:44.754 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b;count:3 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:23:44.755 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: Deploying daemon mon.c on smithi064 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/489364838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 ceph-mon[29477]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 bash[29458]: ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a 2024-02-10T07:23:44.756 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:44 smithi064 systemd[1]: Started Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:23:45.009 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:23:45.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:23:50.043 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:23:50.043 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:23:44.682719Z","created":"2024-02-10T07:22:17.590807Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3301","nonce":0},{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T07:23:50.045 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 2 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: mon.a calling monitor election 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: mon.c calling monitor election 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: fsmap 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: mgrmap e13: y(active, since 36s) 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: overall HEALTH_OK 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:23:50.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: mon.a calling monitor election 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: mon.c calling monitor election 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: fsmap 2024-02-10T07:23:50.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:23:50.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: mgrmap e13: y(active, since 36s) 2024-02-10T07:23:50.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: overall HEALTH_OK 2024-02-10T07:23:50.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:50.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:23:50.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:49 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[24760]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[24760]: Deploying daemon mon.b on smithi136 2024-02-10T07:23:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/2586443335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:51.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[29477]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:51.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[29477]: Deploying daemon mon.b on smithi136 2024-02-10T07:23:51.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/2586443335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:51.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:52.240 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:23:52.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:23:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:52 smithi064 ceph-mon[24760]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:53.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:52 smithi064 ceph-mon[29477]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:54.663 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:23:54.663 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:23:44.682719Z","created":"2024-02-10T07:22:17.590807Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3301","nonce":0},{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T07:23:54.665 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 2 2024-02-10T07:23:54.950 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 podman[25541]: 2024-02-10 07:23:54.95034991 +0000 UTC m=+0.464730273 container init 6cc41643f6204425093dd3680e140016d43d12678e31960b6dbe4d34053ca46b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., release=754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T07:23:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:54 smithi064 ceph-mon[24760]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:23:54 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/1542191025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:54 smithi064 ceph-mon[29477]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:23:54 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/1542191025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:55.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 podman[25541]: 2024-02-10 07:23:54.954120464 +0000 UTC m=+0.468500827 container start 6cc41643f6204425093dd3680e140016d43d12678e31960b6dbe4d34053ca46b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.openshift.tags=base centos centos-stream, release=754, distribution-scope=public, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T07:23:55.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:23:55.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T07:23:55.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pidfile_write: ignore empty --pid-file 2024-02-10T07:23:55.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: load: jerasure load: lrc 2024-02-10T07:23:55.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: RocksDB version: 6.15.5 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Compile date Apr 18 2022 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: DB SUMMARY 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: DB Session ID: A4LZ2YDRZ9WEMTW50IY0 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:23:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.env: 0x564f54ac6860 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.fs: Posix File System 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.info_log: 0x564f55e91dc0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.statistics: (nil) 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.use_fsync: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:23:55.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.db_log_dir: 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.write_buffer_manager: 0x564f55f82240 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:23:55.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.unordered_write: 0 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:23:55.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.row_cache: None 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.wal_filter: None 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.preserve_deletes: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:23:55.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:23:55.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_open_files: -1 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Compression algorithms supported: 2024-02-10T07:23:55.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kZSTD supported: 0 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.merge_operator: 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_filter: None 2024-02-10T07:23:55.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564f55e5dd00) 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_type: 0 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: data_block_index_type: 0 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_shortening: 1 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: hash_index_allow_collision: 1 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: checksum: 1 2024-02-10T07:23:55.233 INFO:journalctl@ceph.mon.b.smithi136.stdout: no_block_cache: 0 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache: 0x564f55ec8170 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_options: 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: capacity : 536870912 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: num_shard_bits : 4 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: strict_capacity_limit : 0 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_compressed: (nil) 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: persistent_cache: (nil) 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_size: 4096 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_size_deviation: 10 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_restart_interval: 16 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_block_restart_interval: 1 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: metadata_block_size: 4096 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: partition_filters: 0 2024-02-10T07:23:55.234 INFO:journalctl@ceph.mon.b.smithi136.stdout: use_delta_encoding: 1 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: whole_key_filtering: 1 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: verify_compression: 0 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: format_version: 4 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_index_compression: 1 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_align: 0 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression: NoCompression 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.num_levels: 7 2024-02-10T07:23:55.235 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:23:55.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:23:55.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:23:55.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.ttl: 2592000 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:23:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549834980991, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549834981299, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707549834, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ac233f77-2f6a-46be-8568-b0e141509236", "db_session_id": "A4LZ2YDRZ9WEMTW50IY0"}} 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549834981596, "job": 1, "event": "recovery_finished"} 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x564f55eaea80 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: DB pointer 0x564f55ebe000 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** DB Stats ** 2024-02-10T07:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.242 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:23:55.243 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:23:55.244 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: using public_addr v2:172.21.15.136:0/0 -> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: starting mon.b rank -1 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(???) e0 preinit fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).mds e1 new map 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: e1 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:23:55.245 INFO:journalctl@ceph.mon.b.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout: legacy client fscid: -1 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout: No filesystems configured 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:23:55.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi136:172.21.15.136=b;count:3 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: Deploying daemon mon.c on smithi064 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/489364838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.a calling monitor election 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.c calling monitor election 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: fsmap 2024-02-10T07:23:55.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mgrmap e13: y(active, since 36s) 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: overall HEALTH_OK 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: Deploying daemon mon.b on smithi136 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/2586443335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/1542191025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:54 smithi136 ceph-mon[25575]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:55 smithi136 bash[25541]: 6cc41643f6204425093dd3680e140016d43d12678e31960b6dbe4d34053ca46b 2024-02-10T07:23:55.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:23:55 smithi136 systemd[1]: Started Ceph mon.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:23:56.609 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:23:56.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: mon.a calling monitor election 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: mon.c calling monitor election 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: mon.b calling monitor election 2024-02-10T07:24:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: fsmap 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: mgrmap e13: y(active, since 47s) 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: overall HEALTH_OK 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:00.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: mon.a calling monitor election 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: mon.c calling monitor election 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: mon.b calling monitor election 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:24:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: fsmap 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: mgrmap e13: y(active, since 47s) 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: overall HEALTH_OK 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: mon.a calling monitor election 2024-02-10T07:24:00.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: mon.c calling monitor election 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: mon.b calling monitor election 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: fsmap 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:24:00.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: mgrmap e13: y(active, since 47s) 2024-02-10T07:24:00.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: overall HEALTH_OK 2024-02-10T07:24:00.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:00.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:00.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:00.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:01.378 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-02-10T07:24:01.378 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":3,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:23:55.001104Z","created":"2024-02-10T07:22:17.590807Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3301","nonce":0},{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-10T07:24:01.380 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 3 2024-02-10T07:24:02.027 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:02.027 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/1872189610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:02.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:02.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:01 smithi136 ceph-mon[25575]: Reconfiguring daemon mon.a on smithi064 2024-02-10T07:24:02.050 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T07:24:02.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/1872189610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[29477]: Reconfiguring daemon mon.a on smithi064 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.310 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/1872189610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:02.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:01 smithi064 ceph-mon[24760]: Reconfiguring daemon mon.a on smithi064 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:04.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[24760]: Reconfiguring daemon mon.c on smithi064 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:04.148 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:03 smithi064 ceph-mon[29477]: Reconfiguring daemon mon.c on smithi064 2024-02-10T07:24:04.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:04.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:04.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:04.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:24:04.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:04.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:04.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:04.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:03 smithi136 ceph-mon[25575]: Reconfiguring daemon mon.c on smithi064 2024-02-10T07:24:04.549 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:24:04.549 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-02-10T07:24:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:24:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-02-10T07:24:05.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:04 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3970652983' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:05.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:04 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3970652983' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:05.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:04 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3970652983' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:05.641 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T07:24:05.641 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:24:05.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T07:24:05.674 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:24:05.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:24:05.742 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:24:05.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T07:24:05.774 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:24:05.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:24:05.845 INFO:tasks.cephadm:Adding mgr.y on smithi064 2024-02-10T07:24:05.845 INFO:tasks.cephadm:Adding mgr.x on smithi136 2024-02-10T07:24:05.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi064=y;smithi136=x' 2024-02-10T07:24:06.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:06 smithi136 ceph-mon[25575]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:06.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:06 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:06.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:06 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:06.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:06 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:06.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:06 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[29477]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[24760]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:07.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:07.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:24:07.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:24:07.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:06 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:07.653 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mgr update... 2024-02-10T07:24:07.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:07 smithi136 ceph-mon[25575]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:24:07.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:07 smithi136 ceph-mon[25575]: Reconfiguring daemon mon.b on smithi136 2024-02-10T07:24:08.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:07 smithi064 ceph-mon[29477]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:24:08.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:07 smithi064 ceph-mon[29477]: Reconfiguring daemon mon.b on smithi136 2024-02-10T07:24:08.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:07 smithi064 ceph-mon[24760]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:24:08.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:07 smithi064 ceph-mon[24760]: Reconfiguring daemon mon.b on smithi136 2024-02-10T07:24:08.752 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:08 smithi136 ceph-mon[25575]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:08.753 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:08 smithi136 ceph-mon[25575]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi136=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:08.753 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:08 smithi136 ceph-mon[25575]: Saving service mgr spec with placement smithi064=y;smithi136=x;count:2 2024-02-10T07:24:08.753 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:08 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:08.776 DEBUG:teuthology.orchestra.run.smithi136:mgr.x> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T07:24:08.778 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T07:24:08.779 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:24:08.779 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T07:24:08.797 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T07:24:08.797 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 07:23:44.121638324 +0000 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 07:21:26.241129937 +0000 2024-02-10T07:24:08.855 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 07:21:26.241129937 +0000 2024-02-10T07:24:08.856 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 07:21:26.241129937 +0000 2024-02-10T07:24:08.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[24760]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[24760]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi136=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[24760]: Saving service mgr spec with placement smithi064=y;smithi136=x;count:2 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[29477]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[29477]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi136=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[29477]: Saving service mgr spec with placement smithi064=y;smithi136=x;count:2 2024-02-10T07:24:08.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:08 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:08.926 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T07:24:08.926 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T07:24:08.926 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000278185 s, 1.8 MB/s 2024-02-10T07:24:08.928 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T07:24:08.985 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-02-10T07:24:09.042 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T07:24:09.042 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:09.042 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T07:24:09.042 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:09.042 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:09.043 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 07:23:44.122638305 +0000 2024-02-10T07:24:09.043 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 07:21:26.610122705 +0000 2024-02-10T07:24:09.043 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 07:21:26.610122705 +0000 2024-02-10T07:24:09.043 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 07:21:26.610122705 +0000 2024-02-10T07:24:09.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T07:24:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T07:24:09.111 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T07:24:09.111 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000203953 s, 2.5 MB/s 2024-02-10T07:24:09.112 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T07:24:09.169 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-02-10T07:24:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T07:24:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 07:23:44.122638305 +0000 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 07:21:26.950116041 +0000 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 07:21:26.950116041 +0000 2024-02-10T07:24:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 07:21:26.950116041 +0000 2024-02-10T07:24:09.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T07:24:09.292 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T07:24:09.292 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T07:24:09.293 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000187724 s, 2.7 MB/s 2024-02-10T07:24:09.294 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T07:24:09.349 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-02-10T07:24:09.404 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T07:24:09.404 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 07:23:44.122638305 +0000 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 07:21:27.292109338 +0000 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 07:21:27.292109338 +0000 2024-02-10T07:24:09.405 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 07:21:27.292109338 +0000 2024-02-10T07:24:09.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T07:24:09.470 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T07:24:09.470 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T07:24:09.470 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000187394 s, 2.7 MB/s 2024-02-10T07:24:09.471 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T07:24:09.527 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:24:09.527 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T07:24:09.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T07:24:09.542 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-02-10T07:24:09.597 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T07:24:09.597 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:09.597 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-10T07:24:09.597 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:09.597 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:09.597 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-02-10 07:23:53.507914682 +0000 2024-02-10T07:24:09.598 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-02-10 07:21:25.092821016 +0000 2024-02-10T07:24:09.598 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-02-10 07:21:25.092821016 +0000 2024-02-10T07:24:09.598 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-02-10 07:21:25.092821016 +0000 2024-02-10T07:24:09.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T07:24:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-02-10T07:24:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-02-10T07:24:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000207889 s, 2.5 MB/s 2024-02-10T07:24:09.665 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T07:24:09.720 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-02-10 07:23:53.508914660 +0000 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-02-10 07:21:25.446812948 +0000 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-02-10 07:21:25.446812948 +0000 2024-02-10T07:24:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-02-10 07:21:25.446812948 +0000 2024-02-10T07:24:09.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T07:24:09.841 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-02-10T07:24:09.841 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-02-10T07:24:09.841 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000175688 s, 2.9 MB/s 2024-02-10T07:24:09.842 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T07:24:09.897 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-02-10T07:24:09.951 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T07:24:09.951 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-02-10 07:23:53.508914660 +0000 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-02-10 07:21:25.790805108 +0000 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-02-10 07:21:25.790805108 +0000 2024-02-10T07:24:09.952 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-02-10 07:21:25.790805108 +0000 2024-02-10T07:24:09.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T07:24:10.017 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-02-10T07:24:10.017 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-02-10T07:24:10.017 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000204032 s, 2.5 MB/s 2024-02-10T07:24:10.018 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T07:24:10.074 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-02-10 07:23:53.508914660 +0000 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-02-10 07:21:26.151796880 +0000 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-02-10 07:21:26.151796880 +0000 2024-02-10T07:24:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-02-10 07:21:26.151796880 +0000 2024-02-10T07:24:10.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T07:24:10.194 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-02-10T07:24:10.194 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-02-10T07:24:10.194 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000178842 s, 2.9 MB/s 2024-02-10T07:24:10.195 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T07:24:10.250 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/vg_nvme/lv_4... 2024-02-10T07:24:10.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T07:24:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[24760]: Deploying daemon mgr.x on smithi136 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:10.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:24:10.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T07:24:10.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:24:10.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:10.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:10 smithi064 ceph-mon[29477]: Deploying daemon mgr.x on smithi136 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:10.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:10 smithi136 ceph-mon[25575]: Deploying daemon mgr.x on smithi136 2024-02-10T07:24:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T07:24:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T07:24:12.492 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T07:24:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T07:24:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115058 s, 911 MB/s 2024-02-10T07:24:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T07:24:12.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[24760]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:12.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:12.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:12.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:12.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:12.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[29477]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:12.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:12.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:12.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:12.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:12.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:12 smithi136 ceph-mon[25575]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:12.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:12.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:12.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:12.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:12.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:12.856+0000 7f71374da000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:24:13.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-02-10T07:24:13.278 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:13.007+0000 7f71374da000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:24:13.278 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:13.087+0000 7f71374da000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:24:13.279 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:13.195+0000 7f71374da000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:24:13.279 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:13.277+0000 7f71374da000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:24:13.657 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:13.438+0000 7f71374da000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:24:13.657 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:13.523+0000 7f71374da000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:24:13.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:24:13.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:24:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[24760]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[29477]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:14 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:14.874 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:14.554+0000 7f71374da000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:24:14.874 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:14.685+0000 7f71374da000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:24:14.874 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:14.873+0000 7f71374da000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:24:14.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-mon[25575]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:14.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:14.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:14.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:14.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:14.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:15.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:14.960+0000 7f71374da000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:24:15.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:15.093+0000 7f71374da000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:24:15.711 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:15.408+0000 7f71374da000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:24:15.711 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:15.625+0000 7f71374da000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:24:15.711 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:15.710+0000 7f71374da000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:24:15.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:15.879+0000 7f71374da000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:24:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:16 smithi136 ceph-mon[25575]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:16 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:24:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:16 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:24:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:16 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:16.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:16.785+0000 7f71374da000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:24:17.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[29477]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:17.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:24:17.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:24:17.033 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:17.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[24760]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:17.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:24:17.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:24:17.033 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:16 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:17.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:17.074+0000 7f71374da000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:24:17.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:17.166+0000 7f71374da000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:24:17.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:24:17.312+0000 7f71374da000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:24:17.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-mon[25575]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:17.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-mon[25575]: Standby manager daemon x started 2024-02-10T07:24:17.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:24:17.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:24:17.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:24:17.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:17 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:24:18.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[24760]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:18.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[24760]: Standby manager daemon x started 2024-02-10T07:24:18.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[29477]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[29477]: Standby manager daemon x started 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:24:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:17 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/740895785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: mgrmap e14: y(active, since 64s), standbys: x 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3805756490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]: dispatch 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]: dispatch 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]': finished 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T07:24:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: mgrmap e14: y(active, since 64s), standbys: x 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3805756490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]: dispatch 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]: dispatch 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]': finished 2024-02-10T07:24:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: mgrmap e14: y(active, since 64s), standbys: x 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3805756490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]: dispatch 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]: dispatch 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5938cc8-fb10-459a-a41c-b568fa16a3f9"}]': finished 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T07:24:19.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:19.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:19 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4241172073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:24:20.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:19 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4241172073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:24:20.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:19 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4241172073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:24:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:20 smithi136 ceph-mon[25575]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:20 smithi064 ceph-mon[24760]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:21.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:20 smithi064 ceph-mon[29477]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:22.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:22 smithi136 ceph-mon[25575]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:23.039 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:22 smithi064 ceph-mon[24760]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:23.040 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:22 smithi064 ceph-mon[29477]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:24 smithi136 ceph-mon[25575]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:24:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:25.025 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:24 smithi064 ceph-mon[24760]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:25.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:24:25.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:25.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:24 smithi064 ceph-mon[29477]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:25.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:24:25.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:25.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:25 smithi064 ceph-mon[29477]: Deploying daemon osd.0 on smithi064 2024-02-10T07:24:25.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:25 smithi064 ceph-mon[24760]: Deploying daemon osd.0 on smithi064 2024-02-10T07:24:25.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:25 smithi136 ceph-mon[25575]: Deploying daemon osd.0 on smithi064 2024-02-10T07:24:26.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:26 smithi064 ceph-mon[24760]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:26.950 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:26 smithi064 ceph-mon[29477]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:26 smithi136 ceph-mon[25575]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:28.824 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:28 smithi064 ceph-mon[24760]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:28.824 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:28 smithi064 ceph-mon[29477]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:28.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:28 smithi136 ceph-mon[25575]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:30 smithi136 ceph-mon[25575]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[24760]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:30.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[29477]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:32.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:32 smithi064 ceph-mon[24760]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:32.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:32 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:32.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:32 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:32.648 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:32 smithi064 ceph-mon[29477]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:32.648 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:32 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:32.648 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:32 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:32.674 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2024-02-10T07:24:32.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:32 smithi136 ceph-mon[25575]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:32.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:32 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:32.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:32 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:33.392 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T07:24:33.394 INFO:tasks.cephadm:Deploying osd.1 on smithi064 with /dev/vg_nvme/lv_3... 2024-02-10T07:24:33.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T07:24:33.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[29477]: from='osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:24:33.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[29477]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:24:33.674 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[24760]: from='osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[24760]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:33.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:33 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:33 smithi136 ceph-mon[25575]: from='osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:24:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:33 smithi136 ceph-mon[25575]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:24:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:33 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:33 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:33 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:33 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:34.660 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[24760]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:34.660 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[24760]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:24:34.661 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[24760]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T07:24:34.661 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:34.661 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[24760]: from='osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:34.661 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[24760]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:34.661 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[29477]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:34.662 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[29477]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:24:34.662 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[29477]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T07:24:34.662 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:34.662 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[29477]: from='osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:34.662 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-mon[29477]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:34.662 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[32329]: 2024-02-10T07:24:34.481+0000 7ff679080700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T07:24:34.663 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:24:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[32329]: 2024-02-10T07:24:34.483+0000 7ff673a17700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:24:34.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:34 smithi136 ceph-mon[25575]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:34 smithi136 ceph-mon[25575]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:24:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:34 smithi136 ceph-mon[25575]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T07:24:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:34 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:34 smithi136 ceph-mon[25575]: from='osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:34 smithi136 ceph-mon[25575]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[24760]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:24:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[24760]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[29477]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[29477]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:35 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:35 smithi136 ceph-mon[25575]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:24:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:35 smithi136 ceph-mon[25575]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T07:24:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:35 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:35 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:35 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[24760]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[24760]: osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601] boot 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[24760]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:24:36.804 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:24:36.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[29477]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:36.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[29477]: osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601] boot 2024-02-10T07:24:36.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[29477]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T07:24:36.805 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:36.954 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T07:24:36.955 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T07:24:36.955 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T07:24:36.955 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T07:24:36.955 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128563 s, 816 MB/s 2024-02-10T07:24:36.955 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T07:24:36.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:36 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:24:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:36 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:24:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:36 smithi136 ceph-mon[25575]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:24:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:36 smithi136 ceph-mon[25575]: osd.0 [v2:172.21.15.64:6802/1428254601,v1:172.21.15.64:6803/1428254601] boot 2024-02-10T07:24:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:36 smithi136 ceph-mon[25575]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T07:24:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:24:37.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-02-10T07:24:37.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[29477]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T07:24:37.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:24:37.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[24760]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T07:24:37.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:24:37.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:37 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:37 smithi136 ceph-mon[25575]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T07:24:37.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:37 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:37 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:24:37.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:37 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:37.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:37 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:38.678 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:38 smithi064 ceph-mon[29477]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:38.679 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:38 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi064 2024-02-10T07:24:38.679 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:38 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi064 to 16064M 2024-02-10T07:24:38.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:38 smithi064 ceph-mon[24760]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:38.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:38 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi064 2024-02-10T07:24:38.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:38 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi064 to 16064M 2024-02-10T07:24:38.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:38 smithi136 ceph-mon[25575]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:38.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:38 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi064 2024-02-10T07:24:38.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:38 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi064 to 16064M 2024-02-10T07:24:40.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[24760]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:40.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[24760]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:40.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:24:40.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:24:40.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:40.656 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[29477]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:40.656 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[29477]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:40.656 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:24:40.656 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:24:40.656 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:40 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:40.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:40 smithi136 ceph-mon[25575]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:40 smithi136 ceph-mon[25575]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:24:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:40 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:24:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:40 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:24:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:40 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:42.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[24760]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1817449038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]: dispatch 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]: dispatch 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]': finished 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[24760]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[29477]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1817449038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]: dispatch 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]: dispatch 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]': finished 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[29477]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T07:24:42.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:42 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:42.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:42 smithi136 ceph-mon[25575]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:42.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:42 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1817449038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]: dispatch 2024-02-10T07:24:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:42 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]: dispatch 2024-02-10T07:24:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:42 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5e6f297-b17d-4a61-98f6-851a3e10b0d4"}]': finished 2024-02-10T07:24:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:42 smithi136 ceph-mon[25575]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T07:24:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:42 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:43.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:43 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2398319689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:24:43.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:43 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2398319689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:24:43.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:43 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2398319689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:24:44.766 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:44 smithi064 ceph-mon[24760]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:44.766 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:44 smithi064 ceph-mon[29477]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:44 smithi136 ceph-mon[25575]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:46.901 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:46 smithi064 ceph-mon[24760]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:46.902 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:46 smithi064 ceph-mon[29477]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:46 smithi136 ceph-mon[25575]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:48.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:48 smithi064 ceph-mon[24760]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:48.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:24:48.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:48.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:48 smithi064 ceph-mon[29477]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:48.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:24:48.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:48.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:48 smithi136 ceph-mon[25575]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:48.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:24:48.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:49.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:49 smithi064 ceph-mon[24760]: Deploying daemon osd.1 on smithi064 2024-02-10T07:24:49.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:49 smithi064 ceph-mon[29477]: Deploying daemon osd.1 on smithi064 2024-02-10T07:24:49.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:49 smithi136 ceph-mon[25575]: Deploying daemon osd.1 on smithi064 2024-02-10T07:24:50.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:50 smithi064 ceph-mon[24760]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:50.765 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:50 smithi064 ceph-mon[29477]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:50 smithi136 ceph-mon[25575]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:52.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:52 smithi064 ceph-mon[24760]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:52.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:52 smithi064 ceph-mon[29477]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:52.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:52 smithi136 ceph-mon[25575]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:54.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[29477]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:54.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:54.298 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:54.298 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:54.298 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:54.298 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[24760]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:54.299 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:54.299 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:54.299 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:54.299 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:53 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:54.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:53 smithi136 ceph-mon[25575]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:53 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:53 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:53 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:53 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:56.287 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 1 on host 'smithi064' 2024-02-10T07:24:56.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:56.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:24:56.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:56.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:56.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:56.943 DEBUG:teuthology.orchestra.run.smithi064:osd.1> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T07:24:56.945 INFO:tasks.cephadm:Deploying osd.2 on smithi064 with /dev/vg_nvme/lv_2... 2024-02-10T07:24:56.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T07:24:56.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:24:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:24:57.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[24760]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:24:57.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[24760]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T07:24:57.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:57.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[24760]: from='osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:57.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[24760]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:57.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[29477]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:24:57.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[29477]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T07:24:57.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:57.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[29477]: from='osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:57.926 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-mon[29477]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:57.926 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[35287]: 2024-02-10T07:24:57.618+0000 7f622b7bc700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T07:24:57.926 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:24:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[35287]: 2024-02-10T07:24:57.620+0000 7f6226153700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:24:57.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:57 smithi136 ceph-mon[25575]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:24:57.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:57 smithi136 ceph-mon[25575]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T07:24:57.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:57 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:57.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:57 smithi136 ceph-mon[25575]: from='osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:57.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:57 smithi136 ceph-mon[25575]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:24:58.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[24760]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[24760]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[24760]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[29477]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[29477]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[29477]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:58.882 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:58 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:58.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:58 smithi136 ceph-mon[25575]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:24:58.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:58 smithi136 ceph-mon[25575]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:24:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:58 smithi136 ceph-mon[25575]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T07:24:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:58 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:58 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:59.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:24:59.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:24:59.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[29477]: osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485] boot 2024-02-10T07:24:59.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[29477]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T07:24:59.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:59.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:24:59.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:24:59.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[24760]: osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485] boot 2024-02-10T07:24:59.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[24760]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T07:24:59.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:24:59 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:24:59.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:59 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:24:59.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:59 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:24:59.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:59 smithi136 ceph-mon[25575]: osd.1 [v2:172.21.15.64:6810/3004829485,v1:172.21.15.64:6811/3004829485] boot 2024-02-10T07:24:59.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:59 smithi136 ceph-mon[25575]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T07:24:59.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:24:59 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:25:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T07:25:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T07:25:00.743 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T07:25:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T07:25:00.743 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112393 s, 933 MB/s 2024-02-10T07:25:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T07:25:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:25:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi064 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi064 to 8032M 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:00 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:25:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T07:25:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi064 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi064 to 8032M 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi064 2024-02-10T07:25:01.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:01.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:01.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi064 to 8032M 2024-02-10T07:25:01.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:00 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:01.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2024-02-10T07:25:02.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:02 smithi064 ceph-mon[24760]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:02.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:02 smithi064 ceph-mon[29477]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:02.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:02 smithi136 ceph-mon[25575]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:03.932 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:03.932 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:03.932 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:03.933 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:03.933 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:03.933 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:04.661 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:04 smithi064 ceph-mon[29477]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:04.661 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:04 smithi064 ceph-mon[29477]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:04.662 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:04 smithi064 ceph-mon[24760]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:04.662 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:04 smithi064 ceph-mon[24760]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:04.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:04 smithi136 ceph-mon[25575]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:04 smithi136 ceph-mon[25575]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:05.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:05 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/975141890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]: dispatch 2024-02-10T07:25:05.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:05 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]: dispatch 2024-02-10T07:25:05.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:05 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]': finished 2024-02-10T07:25:05.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:05 smithi136 ceph-mon[25575]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T07:25:05.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:05 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:06.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/975141890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]: dispatch 2024-02-10T07:25:06.018 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]: dispatch 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]': finished 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[24760]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/975141890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]: dispatch 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]: dispatch 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5edca33-a497-4211-a1ab-1c0e448c2b2c"}]': finished 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[29477]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T07:25:06.019 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:05 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:06.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:06 smithi064 ceph-mon[24760]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:06.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:06 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1202658918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:06.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:06 smithi064 ceph-mon[29477]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:06.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:06 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1202658918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:06.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:06 smithi136 ceph-mon[25575]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:06.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:06 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1202658918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:08.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:08 smithi064 ceph-mon[24760]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:08.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:08 smithi064 ceph-mon[29477]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:08.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:08 smithi136 ceph-mon[25575]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:10.937 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:10 smithi064 ceph-mon[29477]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:10.937 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:10 smithi064 ceph-mon[24760]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:10 smithi136 ceph-mon[25575]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:12.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:12 smithi136 ceph-mon[25575]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:12.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:25:12.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:12 smithi064 ceph-mon[29477]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:25:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:12 smithi064 ceph-mon[24760]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:25:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:13.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:13 smithi064 ceph-mon[29477]: Deploying daemon osd.2 on smithi064 2024-02-10T07:25:13.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:25:13.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:25:13.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:13 smithi064 ceph-mon[24760]: Deploying daemon osd.2 on smithi064 2024-02-10T07:25:13.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:25:13.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:25:13.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:13 smithi136 ceph-mon[25575]: Deploying daemon osd.2 on smithi064 2024-02-10T07:25:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:25:13.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:25:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:14 smithi064 ceph-mon[24760]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:14.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:14 smithi064 ceph-mon[29477]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:14.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:14 smithi136 ceph-mon[25575]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:16.799 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:16 smithi064 ceph-mon[24760]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:16.799 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:16 smithi064 ceph-mon[29477]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:16 smithi136 ceph-mon[25575]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:18.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[24760]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:18.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:18.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:18.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:18.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:18.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[29477]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:18.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:18.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:18.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:18.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:18.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:18 smithi136 ceph-mon[25575]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:18.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:18.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:18.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:18.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:20.333 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 2 on host 'smithi064' 2024-02-10T07:25:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:25:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:25:20.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:20.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:20.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:20 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:20.955 DEBUG:teuthology.orchestra.run.smithi064:osd.2> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T07:25:20.958 INFO:tasks.cephadm:Deploying osd.3 on smithi064 with /dev/vg_nvme/lv_1... 2024-02-10T07:25:20.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T07:25:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:25:20.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:25:20.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:20.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:20.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:20.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:20 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[24760]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:25:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[24760]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T07:25:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[24760]: from='osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[24760]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[29477]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:25:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[29477]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T07:25:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[29477]: from='osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-mon[29477]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:21.815 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[38216]: 2024-02-10T07:25:21.621+0000 7f2a76499700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T07:25:21.815 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:21 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[38216]: 2024-02-10T07:25:21.624+0000 7f2a6f62d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:25:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:21 smithi136 ceph-mon[25575]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:25:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:21 smithi136 ceph-mon[25575]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T07:25:21.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:21 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:21.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:21 smithi136 ceph-mon[25575]: from='osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:21.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:21 smithi136 ceph-mon[25575]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:22.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[24760]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:22.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[24760]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:25:22.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[24760]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T07:25:22.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:22.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:22.751 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[29477]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:22.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[29477]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:25:22.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[29477]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T07:25:22.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:22.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:22.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:22 smithi136 ceph-mon[25575]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:22.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:22 smithi136 ceph-mon[25575]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:25:22.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:22 smithi136 ceph-mon[25575]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T07:25:22.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:22.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:23.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:25:23.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:25:23.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[24760]: osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094] boot 2024-02-10T07:25:23.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[24760]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T07:25:23.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:23.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T07:25:23.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:25:23.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:25:23.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[29477]: osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094] boot 2024-02-10T07:25:23.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[29477]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T07:25:23.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:23.807 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:23 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T07:25:23.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:23 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:25:23.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:23 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:25:23.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:23 smithi136 ceph-mon[25575]: osd.2 [v2:172.21.15.64:6818/2748450094,v1:172.21.15.64:6819/2748450094] boot 2024-02-10T07:25:23.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:23 smithi136 ceph-mon[25575]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T07:25:23.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:23 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:25:23.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:23 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T07:25:24.849 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T07:25:24.849 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T07:25:24.850 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T07:25:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T07:25:24.850 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113867 s, 921 MB/s 2024-02-10T07:25:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T07:25:24.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:24.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:24 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:24 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:25.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2024-02-10T07:25:25.788 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi064 2024-02-10T07:25:25.788 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi064 to 5354M 2024-02-10T07:25:25.788 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:25:25.788 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[29477]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T07:25:25.788 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi064 2024-02-10T07:25:25.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi064 to 5354M 2024-02-10T07:25:25.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:25:25.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:25 smithi064 ceph-mon[24760]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T07:25:25.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:25 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi064 2024-02-10T07:25:25.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:25 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi064 to 5354M 2024-02-10T07:25:25.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:25 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:25:25.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:25 smithi136 ceph-mon[25575]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T07:25:26.060 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:25:25 smithi064 sudo[39485]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:25:26.060 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:25:25 smithi064 sudo[39485]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:26.060 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:25:25 smithi064 sudo[39485]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:26.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39506]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:25:26.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39506]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:26.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39506]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:26.637 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39512]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:25:26.637 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39512]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:26.637 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39512]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:26.896 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[24760]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:26.896 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[24760]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T07:25:26.896 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[24760]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T07:25:26.896 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:26.896 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[29477]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:26.896 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[29477]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T07:25:26.897 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[29477]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T07:25:26.897 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:26 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:26.897 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39515]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:25:26.897 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39515]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:26.897 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39515]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:26 smithi136 ceph-mon[25575]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:26 smithi136 ceph-mon[25575]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T07:25:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:26 smithi136 ceph-mon[25575]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T07:25:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:26 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:27.167 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39536]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:25:27.168 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39536]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:27.168 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:26 smithi064 sudo[39536]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:27.424 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39544]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:25:27.424 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39544]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:27.424 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39544]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:27.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39569]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:25:27.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39569]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:27.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39569]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:27.945 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39573]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:25:27.945 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39573]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:27.945 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39573]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:28.213 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39633]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:25:28.213 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39633]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:28.213 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:27 smithi064 sudo[39633]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:28.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:28.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:28.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:28.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:28.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 sudo[27086]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:25:28.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 sudo[27086]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:28.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 sudo[27086]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:28.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:28.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:28.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 sudo[39659]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 sudo[39659]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 sudo[39659]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:28.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:28.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 sudo[27089]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:25:28.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 sudo[27089]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:25:28.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:28 smithi136 sudo[27089]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:25:29.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:29.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:29.335 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:29.336 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:29 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:29.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:25:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:29 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[29477]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/524033287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e28605d6-1761-45da-813c-e41956e65e01"}]: dispatch 2024-02-10T07:25:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/524033287' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e28605d6-1761-45da-813c-e41956e65e01"}]': finished 2024-02-10T07:25:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[29477]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T07:25:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2957473882' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[24760]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/524033287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e28605d6-1761-45da-813c-e41956e65e01"}]: dispatch 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/524033287' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e28605d6-1761-45da-813c-e41956e65e01"}]': finished 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[24760]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:30 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2957473882' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:31.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:30 smithi136 ceph-mon[25575]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:31.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:30 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/524033287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e28605d6-1761-45da-813c-e41956e65e01"}]: dispatch 2024-02-10T07:25:31.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:30 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/524033287' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e28605d6-1761-45da-813c-e41956e65e01"}]': finished 2024-02-10T07:25:31.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:30 smithi136 ceph-mon[25575]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T07:25:31.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:31.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:30 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2957473882' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:32 smithi064 ceph-mon[24760]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:33.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:32 smithi064 ceph-mon[29477]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:33.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:32 smithi136 ceph-mon[25575]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:34.872 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:34 smithi064 ceph-mon[24760]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:34.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:34 smithi064 ceph-mon[29477]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:35.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:34 smithi136 ceph-mon[25575]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:36.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:36 smithi064 ceph-mon[24760]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:36.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:25:36.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:36.860 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:36 smithi064 ceph-mon[29477]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:36.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:25:36.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:37.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:36 smithi136 ceph-mon[25575]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:25:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:37.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:37 smithi064 ceph-mon[24760]: Deploying daemon osd.3 on smithi064 2024-02-10T07:25:37.790 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:37 smithi064 ceph-mon[29477]: Deploying daemon osd.3 on smithi064 2024-02-10T07:25:38.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:37 smithi136 ceph-mon[25575]: Deploying daemon osd.3 on smithi064 2024-02-10T07:25:39.059 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:38 smithi064 ceph-mon[24760]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:38 smithi064 ceph-mon[29477]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:39.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:38 smithi136 ceph-mon[25575]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:41.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:40 smithi064 ceph-mon[24760]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:41.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:40 smithi064 ceph-mon[29477]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:41.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:40 smithi136 ceph-mon[25575]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:42.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[24760]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:42.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:42.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[29477]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:42.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:42 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:42.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:42 smithi136 ceph-mon[25575]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:42.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:42 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:42.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:42 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:42.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:42 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:42.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:42 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:44.531 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:44 smithi064 ceph-mon[24760]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:44.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:44 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:44.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:44 smithi064 ceph-mon[29477]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:44.532 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:44 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:44.560 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 3 on host 'smithi064' 2024-02-10T07:25:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:44 smithi136 ceph-mon[25575]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:44 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.290 DEBUG:teuthology.orchestra.run.smithi064:osd.3> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T07:25:45.293 INFO:tasks.cephadm:Deploying osd.4 on smithi136 with /dev/vg_nvme/lv_4... 2024-02-10T07:25:45.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:45.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:45.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:45.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:25:45.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:25:45.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:45.605 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:25:45.605 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:45.605 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:25:46.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[29477]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[29477]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[29477]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[29477]: from='osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[29477]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[24760]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[24760]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[24760]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T07:25:46.509 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:46.510 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[24760]: from='osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:46.510 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-mon[24760]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:46.510 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[41193]: 2024-02-10T07:25:46.259+0000 7f77757bc700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T07:25:46.510 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:25:46 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[41193]: 2024-02-10T07:25:46.264+0000 7f7770153700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:46 smithi136 ceph-mon[25575]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:46 smithi136 ceph-mon[25575]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:46 smithi136 ceph-mon[25575]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:46 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:46 smithi136 ceph-mon[25575]: from='osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:46 smithi136 ceph-mon[25575]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:25:47.283 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T07:25:47.283 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T07:25:47.283 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-02-10T07:25:47.284 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-02-10T07:25:47.284 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00866539 s, 1.2 GB/s 2024-02-10T07:25:47.284 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-02-10T07:25:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[24760]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:25:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[24760]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T07:25:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:47.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[29477]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:25:47.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[29477]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T07:25:47.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:47.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:47 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:47.634 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:47 smithi136 ceph-mon[25575]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T07:25:47.634 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:47 smithi136 ceph-mon[25575]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T07:25:47.634 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:47 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:47.635 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:47 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:47.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-02-10T07:25:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:25:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:25:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016] boot 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:25:48.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016] boot 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:48 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.606 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:25:48.606 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: osd.3 [v2:172.21.15.64:6826/2011470016,v1:172.21.15.64:6827/2011470016] boot 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.607 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:25:48.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:48.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:48 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:25:49.513 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:49 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi064 2024-02-10T07:25:49.514 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:49 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi064 to 4016M 2024-02-10T07:25:49.514 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:49 smithi136 ceph-mon[25575]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T07:25:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:49 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi064 2024-02-10T07:25:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:49 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi064 to 4016M 2024-02-10T07:25:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:49 smithi064 ceph-mon[29477]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T07:25:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:49 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi064 2024-02-10T07:25:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:49 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi064 to 4016M 2024-02-10T07:25:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:49 smithi064 ceph-mon[24760]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T07:25:50.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:50 smithi136 ceph-mon[25575]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:50.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:50.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:50.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[29477]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[24760]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:25:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:25:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:51.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:51 smithi136 ceph-mon[25575]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:51 smithi064 ceph-mon[24760]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:52.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:51 smithi064 ceph-mon[29477]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:25:52.883 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:52 smithi136 ceph-mon[25575]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:52.883 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:52 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/2588048236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]: dispatch 2024-02-10T07:25:52.883 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:52 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]: dispatch 2024-02-10T07:25:52.883 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:52 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]': finished 2024-02-10T07:25:52.883 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:52 smithi136 ceph-mon[25575]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T07:25:52.883 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:52 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[24760]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/2588048236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]: dispatch 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]: dispatch 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]': finished 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[24760]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:25:52.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[29477]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:52.939 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/2588048236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]: dispatch 2024-02-10T07:25:52.939 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]: dispatch 2024-02-10T07:25:52.939 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b71b3cb7-13dc-400e-90a3-40212186dd8c"}]': finished 2024-02-10T07:25:52.939 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[29477]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T07:25:52.939 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:52 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:25:53.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:53 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/2346217611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:53 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/2346217611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:54.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:53 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/2346217611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:25:54.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:54 smithi136 ceph-mon[25575]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:54 smithi064 ceph-mon[24760]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:55.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:54 smithi064 ceph-mon[29477]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:56 smithi136 ceph-mon[25575]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:56 smithi064 ceph-mon[24760]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:57.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:56 smithi064 ceph-mon[29477]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:58.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:58 smithi136 ceph-mon[25575]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:58.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:58 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:25:58.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:58 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:59.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:58 smithi064 ceph-mon[24760]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:59.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:58 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:25:59.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:58 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:59.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:58 smithi064 ceph-mon[29477]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:25:59.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:58 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:25:59.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:58 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:25:59.759 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:25:59 smithi136 ceph-mon[25575]: Deploying daemon osd.4 on smithi136 2024-02-10T07:26:00.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:25:59 smithi064 ceph-mon[29477]: Deploying daemon osd.4 on smithi136 2024-02-10T07:26:00.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:25:59 smithi064 ceph-mon[24760]: Deploying daemon osd.4 on smithi136 2024-02-10T07:26:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:00 smithi136 ceph-mon[25575]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:00 smithi064 ceph-mon[24760]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:01.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:00 smithi064 ceph-mon[29477]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:02.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:02 smithi136 ceph-mon[25575]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:02 smithi064 ceph-mon[24760]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:03.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:02 smithi064 ceph-mon[29477]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:03.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:03 smithi136 ceph-mon[25575]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:03.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:04.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[24760]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:04.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:04.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:04.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:04.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:04.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[29477]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:04.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:04.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:04.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:04.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:05.839 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 4 on host 'smithi136' 2024-02-10T07:26:06.398 DEBUG:teuthology.orchestra.run.smithi136:osd.4> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T07:26:06.401 INFO:tasks.cephadm:Deploying osd.5 on smithi136 with /dev/vg_nvme/lv_3... 2024-02-10T07:26:06.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T07:26:07.079 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:06 smithi136 ceph-mon[25575]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:07.079 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:06 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:07.079 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:06 smithi136 ceph-mon[25575]: from='osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:26:07.079 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:06 smithi136 ceph-mon[25575]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:26:07.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[24760]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:07.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:07.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[24760]: from='osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:26:07.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[24760]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:26:07.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[29477]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:07.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:07.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[29477]: from='osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:26:07.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:06 smithi064 ceph-mon[29477]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: from='osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:08.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:08.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[28936]: 2024-02-10T07:26:07.861+0000 7f70342de700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T07:26:08.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:26:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[28936]: 2024-02-10T07:26:07.865+0000 7f7030478700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:26:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:26:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T07:26:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: from='osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: from='osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:07 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:08.548 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T07:26:08.548 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T07:26:08.548 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-02-10T07:26:08.548 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-02-10T07:26:08.549 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0099822 s, 1.1 GB/s 2024-02-10T07:26:08.549 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-02-10T07:26:09.178 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:08 smithi136 ceph-mon[25575]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:09.178 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:08 smithi136 ceph-mon[25575]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T07:26:09.178 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:08 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:09.178 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:08 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:09.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[24760]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:09.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[24760]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T07:26:09.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:09.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:09.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[29477]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:09.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[29477]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T07:26:09.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:09.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:08 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:09.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520] boot 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:10.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:09 smithi136 ceph-mon[25575]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T07:26:10.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:26:10.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:26:10.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520] boot 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[24760]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: osd.4 [v2:172.21.15.136:6800/2071893520,v1:172.21.15.136:6801/2071893520] boot 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:26:10.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:09 smithi064 ceph-mon[29477]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T07:26:11.929 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:11 smithi136 ceph-mon[25575]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T07:26:12.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:11 smithi064 ceph-mon[24760]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T07:26:12.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:11 smithi064 ceph-mon[29477]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:12.877 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:12.878 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:12.878 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:12 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T07:26:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:13.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:12 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:13.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:13 smithi136 ceph-mon[25575]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:13.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:13 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi136 2024-02-10T07:26:13.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:13 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi136 to 17088M 2024-02-10T07:26:13.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:13 smithi136 ceph-mon[25575]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T07:26:13.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:26:13.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:26:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[24760]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi136 2024-02-10T07:26:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi136 to 17088M 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[24760]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[29477]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi136 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi136 to 17088M 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[29477]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:26:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:26:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:14 smithi136 ceph-mon[25575]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:14 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/1820313346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]: dispatch 2024-02-10T07:26:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:14 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]: dispatch 2024-02-10T07:26:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:14 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]': finished 2024-02-10T07:26:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:14 smithi136 ceph-mon[25575]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T07:26:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:14 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[24760]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/1820313346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]: dispatch 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]: dispatch 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]': finished 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[24760]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[29477]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/1820313346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]: dispatch 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]: dispatch 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a87215f6-b76a-4354-b969-8a6d6cb9c5a6"}]': finished 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[29477]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T07:26:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:14 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:15.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:15 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/3711161657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:26:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:15 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/3711161657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:26:16.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:15 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/3711161657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:26:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:16 smithi136 ceph-mon[25575]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-02-10T07:26:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:16 smithi064 ceph-mon[24760]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-02-10T07:26:17.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:16 smithi064 ceph-mon[29477]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-02-10T07:26:18.825 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:18 smithi136 ceph-mon[25575]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T07:26:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:18 smithi064 ceph-mon[24760]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T07:26:19.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:18 smithi064 ceph-mon[29477]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T07:26:20.854 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:20 smithi136 ceph-mon[25575]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:26:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:20 smithi064 ceph-mon[24760]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:26:21.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:20 smithi064 ceph-mon[29477]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:26:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:21 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:26:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:21 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:22.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:21 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:26:22.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:21 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:22.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:21 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:26:22.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:21 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:22.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:22 smithi136 ceph-mon[25575]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-10T07:26:22.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:22 smithi136 ceph-mon[25575]: Deploying daemon osd.5 on smithi136 2024-02-10T07:26:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:22 smithi064 ceph-mon[24760]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-10T07:26:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:22 smithi064 ceph-mon[24760]: Deploying daemon osd.5 on smithi136 2024-02-10T07:26:23.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:22 smithi064 ceph-mon[29477]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-10T07:26:23.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:22 smithi064 ceph-mon[29477]: Deploying daemon osd.5 on smithi136 2024-02-10T07:26:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:24 smithi136 ceph-mon[25575]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:26:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:24 smithi064 ceph-mon[24760]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:26:25.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:24 smithi064 ceph-mon[29477]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:26:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:26 smithi136 ceph-mon[25575]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:26 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:26 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:26 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:26.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:26 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[24760]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[29477]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:27.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:27.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:26 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:28.649 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 5 on host 'smithi136' 2024-02-10T07:26:28.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:28 smithi136 ceph-mon[25575]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:28.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:29.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:28 smithi064 ceph-mon[24760]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:29.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:29.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:28 smithi064 ceph-mon[29477]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:29.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:29.234 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T07:26:29.237 INFO:tasks.cephadm:Deploying osd.6 on smithi136 with /dev/vg_nvme/lv_2... 2024-02-10T07:26:29.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T07:26:29.942 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:29 smithi136 ceph-mon[25575]: from='osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:26:29.942 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:29 smithi136 ceph-mon[25575]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:26:30.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:29 smithi064 ceph-mon[24760]: from='osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:26:30.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:29 smithi064 ceph-mon[24760]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:26:30.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:29 smithi064 ceph-mon[29477]: from='osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:26:30.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:29 smithi064 ceph-mon[29477]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:26:30.943 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:30.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:26:30.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T07:26:30.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:30.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: from='osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:30.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:30.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:30.944 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[31871]: 2024-02-10T07:26:30.702+0000 7f3a10310700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T07:26:30.944 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:26:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[31871]: 2024-02-10T07:26:30.707+0000 7f3a0ccab700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:26:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:26:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T07:26:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: from='osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: from='osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:30 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:31.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:31 smithi136 ceph-mon[25575]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:31 smithi136 ceph-mon[25575]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T07:26:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:31 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:31 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.059 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T07:26:32.059 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T07:26:32.059 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-02-10T07:26:32.060 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-02-10T07:26:32.060 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112991 s, 928 MB/s 2024-02-10T07:26:32.060 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-02-10T07:26:32.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[24760]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:32.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[24760]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T07:26:32.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[29477]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:32.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[29477]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T07:26:32.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:31 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.933 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:26:32.933 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:26:32.933 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:32.933 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.933 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341] boot 2024-02-10T07:26:32.933 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T07:26:32.934 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:32 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:32.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-02-10T07:26:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:26:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:26:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341] boot 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: osd.5 [v2:172.21.15.136:6808/1778300341,v1:172.21.15.136:6809/1778300341] boot 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T07:26:33.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:32 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:26:33.781 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:33 smithi136 ceph-mon[25575]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T07:26:34.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:33 smithi064 ceph-mon[24760]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T07:26:34.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:33 smithi064 ceph-mon[29477]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T07:26:34.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:34 smithi136 ceph-mon[25575]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:34 smithi136 ceph-mon[25575]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:26:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:34 smithi136 ceph-mon[25575]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T07:26:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:34 smithi136 ceph-mon[25575]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T07:26:35.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[24760]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:35.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[24760]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:26:35.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[24760]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T07:26:35.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[24760]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T07:26:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[29477]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[29477]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:26:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[29477]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T07:26:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:34 smithi064 ceph-mon[29477]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T07:26:36.284 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:36.284 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi136 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi136 to 8544M 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:36.285 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:36.286 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:36 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:36.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:36.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi136 2024-02-10T07:26:36.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi136 to 8544M 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:36.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi136 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi136 to 8544M 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:36.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:36.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:36.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:36 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:38.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[24760]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:38.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/4014551623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]: dispatch 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]: dispatch 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]': finished 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[24760]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[29477]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/4014551623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]: dispatch 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]: dispatch 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]': finished 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[29477]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T07:26:38.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:38 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:38.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:38 smithi136 ceph-mon[25575]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:38.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:38 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/4014551623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]: dispatch 2024-02-10T07:26:38.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:38 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]: dispatch 2024-02-10T07:26:38.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:38 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6cc79b12-52f6-409c-873d-ff5537c2e260"}]': finished 2024-02-10T07:26:38.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:38 smithi136 ceph-mon[25575]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T07:26:38.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:38 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:39 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/1488316851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:26:39.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:39 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/1488316851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:26:39.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:39 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/1488316851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:26:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:40 smithi064 ceph-mon[24760]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-02-10T07:26:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:40 smithi064 ceph-mon[24760]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T07:26:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:40 smithi064 ceph-mon[24760]: Cluster is now healthy 2024-02-10T07:26:40.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:40 smithi064 ceph-mon[29477]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-02-10T07:26:40.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:40 smithi064 ceph-mon[29477]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T07:26:40.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:40 smithi064 ceph-mon[29477]: Cluster is now healthy 2024-02-10T07:26:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:40 smithi136 ceph-mon[25575]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-02-10T07:26:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:40 smithi136 ceph-mon[25575]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T07:26:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:40 smithi136 ceph-mon[25575]: Cluster is now healthy 2024-02-10T07:26:42.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:42 smithi136 ceph-mon[25575]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-02-10T07:26:42.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:42 smithi064 ceph-mon[24760]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-02-10T07:26:42.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:42 smithi064 ceph-mon[29477]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-02-10T07:26:44.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:44 smithi136 ceph-mon[25575]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:26:44.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:44 smithi064 ceph-mon[24760]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:26:44.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:44 smithi064 ceph-mon[29477]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:26:45.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:26:45.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:45 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:45.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:45 smithi064 ceph-mon[24760]: Deploying daemon osd.6 on smithi136 2024-02-10T07:26:45.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:26:45.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:45 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:45.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:45 smithi064 ceph-mon[29477]: Deploying daemon osd.6 on smithi136 2024-02-10T07:26:45.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:26:45.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:45 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:45.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:45 smithi136 ceph-mon[25575]: Deploying daemon osd.6 on smithi136 2024-02-10T07:26:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:46 smithi136 ceph-mon[25575]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-02-10T07:26:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:46 smithi064 ceph-mon[24760]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-02-10T07:26:47.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:46 smithi064 ceph-mon[29477]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-02-10T07:26:48.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:48 smithi136 ceph-mon[25575]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:26:49.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:48 smithi064 ceph-mon[24760]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:26:49.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:48 smithi064 ceph-mon[29477]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:26:50.719 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:50 smithi136 ceph-mon[25575]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:50.719 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:50.719 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:50.719 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:50.719 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:50 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[24760]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[29477]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:50.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:50.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:50.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:50.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:50 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:26:52.334 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 6 on host 'smithi136' 2024-02-10T07:26:52.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:52 smithi136 ceph-mon[25575]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:52.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:52 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:52 smithi064 ceph-mon[24760]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:52 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:52.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:52 smithi064 ceph-mon[29477]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:52.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:52 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:53.003 DEBUG:teuthology.orchestra.run.smithi136:osd.6> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T07:26:53.006 INFO:tasks.cephadm:Deploying osd.7 on smithi136 with /dev/vg_nvme/lv_1... 2024-02-10T07:26:53.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T07:26:53.618 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:53 smithi136 ceph-mon[25575]: from='osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:26:53.618 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:53 smithi136 ceph-mon[25575]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:26:53.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:53 smithi064 ceph-mon[24760]: from='osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:26:53.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:53 smithi064 ceph-mon[24760]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:26:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:53 smithi064 ceph-mon[29477]: from='osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:26:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:53 smithi064 ceph-mon[29477]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:26:54.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-mon[25575]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:54.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-mon[25575]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:26:54.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-mon[25575]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T07:26:54.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:54.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-mon[25575]: from='osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:54.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-mon[25575]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:54.622 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[34763]: 2024-02-10T07:26:54.364+0000 7f5047e1e700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T07:26:54.623 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:26:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[34763]: 2024-02-10T07:26:54.370+0000 7f5042fb6700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:26:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[24760]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[24760]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:26:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[24760]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T07:26:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[24760]: from='osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[24760]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[29477]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[29477]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[29477]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[29477]: from='osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:54 smithi064 ceph-mon[29477]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:26:55.236 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T07:26:55.237 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T07:26:55.237 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-02-10T07:26:55.237 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-02-10T07:26:55.237 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0105489 s, 994 MB/s 2024-02-10T07:26:55.237 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-02-10T07:26:55.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:55 smithi136 ceph-mon[25575]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:55.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:55 smithi136 ceph-mon[25575]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T07:26:55.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:55 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:55.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:55 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:55.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:55 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[24760]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[24760]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T07:26:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:55.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[29477]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:26:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[29477]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T07:26:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:55 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:26:55.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-02-10T07:26:56.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:56 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:26:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:56 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:26:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:56 smithi136 ceph-mon[25575]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:56 smithi136 ceph-mon[25575]: osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804] boot 2024-02-10T07:26:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:56 smithi136 ceph-mon[25575]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T07:26:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:56 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:26:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:26:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[24760]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[24760]: osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804] boot 2024-02-10T07:26:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[24760]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T07:26:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:26:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:26:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[29477]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:26:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[29477]: osd.6 [v2:172.21.15.136:6816/3781698804,v1:172.21.15.136:6817/3781698804] boot 2024-02-10T07:26:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[29477]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T07:26:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:56 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:26:57.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:57 smithi136 ceph-mon[25575]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T07:26:57.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:57 smithi064 ceph-mon[29477]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T07:26:57.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:57 smithi064 ceph-mon[24760]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T07:26:58.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:58 smithi136 ceph-mon[25575]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:26:58.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:58 smithi136 ceph-mon[25575]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T07:26:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:58 smithi064 ceph-mon[24760]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:26:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:58 smithi064 ceph-mon[24760]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T07:26:58.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:58 smithi064 ceph-mon[29477]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:26:58.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:58 smithi064 ceph-mon[29477]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T07:26:59.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:59 smithi136 ceph-mon[25575]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T07:26:59.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:59 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:59.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:59 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:59.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:26:59 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:59.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[24760]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T07:26:59.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:59.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:59.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:59.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[29477]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T07:26:59.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:26:59.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:26:59.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:26:59 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:00 smithi136 ceph-mon[25575]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:00 smithi136 ceph-mon[25575]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:00 smithi136 ceph-mon[25575]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T07:27:00.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:00 smithi064 ceph-mon[24760]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:00.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:00 smithi064 ceph-mon[24760]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:00.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:00 smithi064 ceph-mon[24760]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T07:27:00.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:00 smithi064 ceph-mon[29477]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:00.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:00 smithi064 ceph-mon[29477]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:00.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:00 smithi064 ceph-mon[29477]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi136 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi136 to 5696M 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:02.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:02.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:02 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:02.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi136 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi136 to 5696M 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi136 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi136 to 5696M 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:02.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:02 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:03.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:03 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/4168493662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]: dispatch 2024-02-10T07:27:03.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:03 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]: dispatch 2024-02-10T07:27:03.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:03 smithi136 ceph-mon[25575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]': finished 2024-02-10T07:27:03.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:03 smithi136 ceph-mon[25575]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T07:27:03.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:03 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:03.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:03 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/1041092012' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:27:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/4168493662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]: dispatch 2024-02-10T07:27:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]: dispatch 2024-02-10T07:27:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[24760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]': finished 2024-02-10T07:27:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[24760]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T07:27:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/1041092012' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:27:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/4168493662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]: dispatch 2024-02-10T07:27:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]: dispatch 2024-02-10T07:27:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[29477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6369935e-1d08-4497-8930-19f4d84f7291"}]': finished 2024-02-10T07:27:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[29477]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T07:27:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:03 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/1041092012' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:27:04.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:04 smithi136 ceph-mon[25575]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T07:27:04.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:04 smithi064 ceph-mon[24760]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T07:27:04.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:04 smithi064 ceph-mon[29477]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T07:27:06.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:06 smithi136 ceph-mon[25575]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-10T07:27:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:06 smithi064 ceph-mon[24760]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-10T07:27:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:06 smithi064 ceph-mon[29477]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-10T07:27:08.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:08 smithi136 ceph-mon[25575]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:27:08.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:08 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:27:08.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:08 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:08 smithi064 ceph-mon[24760]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:27:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:08 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:27:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:08 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:09.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:08 smithi064 ceph-mon[29477]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:27:09.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:08 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:27:09.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:08 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:09.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:09 smithi136 ceph-mon[25575]: Deploying daemon osd.7 on smithi136 2024-02-10T07:27:10.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:09 smithi064 ceph-mon[24760]: Deploying daemon osd.7 on smithi136 2024-02-10T07:27:10.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:09 smithi064 ceph-mon[29477]: Deploying daemon osd.7 on smithi136 2024-02-10T07:27:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:10 smithi136 ceph-mon[25575]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-02-10T07:27:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:10 smithi064 ceph-mon[24760]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-02-10T07:27:11.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:10 smithi064 ceph-mon[29477]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-02-10T07:27:12.955 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:12 smithi136 ceph-mon[25575]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:27:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:12 smithi064 ceph-mon[24760]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:27:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:12 smithi064 ceph-mon[29477]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T07:27:13.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:13.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:13.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:13.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:13.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:13.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:13 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:14.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:14.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:13 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:14.914 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:14 smithi136 ceph-mon[25575]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:14 smithi064 ceph-mon[24760]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:15.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:14 smithi064 ceph-mon[29477]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:16.681 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 7 on host 'smithi136' 2024-02-10T07:27:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:16 smithi136 ceph-mon[25575]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:16 smithi136 ceph-mon[25575]: from='osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:27:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:16 smithi136 ceph-mon[25575]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:27:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:16 smithi064 ceph-mon[24760]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:16 smithi064 ceph-mon[24760]: from='osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:27:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:16 smithi064 ceph-mon[24760]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:27:17.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:16 smithi064 ceph-mon[29477]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:17.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:16 smithi064 ceph-mon[29477]: from='osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:27:17.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:16 smithi064 ceph-mon[29477]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:27:17.334 DEBUG:teuthology.orchestra.run.smithi136:osd.7> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T07:27:17.336 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T07:27:17.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T07:27:17.865 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[37671]: 2024-02-10T07:27:17.671+0000 7f59e83e0700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T07:27:17.865 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[37671]: 2024-02-10T07:27:17.679+0000 7f59e0d73700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:27:17.865 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:27:17.865 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T07:27:17.865 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:27:17.865 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:17 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:27:17.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T07:27:17.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:17.990 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:17 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:18.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:18 smithi136 ceph-mon[25575]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:18.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:18 smithi136 ceph-mon[25575]: osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306] boot 2024-02-10T07:27:18.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:18 smithi136 ceph-mon[25575]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T07:27:18.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:18 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[24760]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[24760]: osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306] boot 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[24760]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[29477]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[29477]: osd.7 [v2:172.21.15.136:6824/4091613306,v1:172.21.15.136:6825/4091613306] boot 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[29477]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T07:27:19.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:18 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:27:19.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:19 smithi136 ceph-mon[25575]: purged_snaps scrub starts 2024-02-10T07:27:19.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:19 smithi136 ceph-mon[25575]: purged_snaps scrub ok 2024-02-10T07:27:19.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:19 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/151924281' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T07:27:19.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:19 smithi136 ceph-mon[25575]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T07:27:20.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[24760]: purged_snaps scrub starts 2024-02-10T07:27:20.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[24760]: purged_snaps scrub ok 2024-02-10T07:27:20.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/151924281' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T07:27:20.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[24760]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T07:27:20.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[29477]: purged_snaps scrub starts 2024-02-10T07:27:20.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[29477]: purged_snaps scrub ok 2024-02-10T07:27:20.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/151924281' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T07:27:20.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:19 smithi064 ceph-mon[29477]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T07:27:20.127 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1707550038,"num_in_osds":8,"osd_in_since":1707550022,"num_remapped_pgs":1} 2024-02-10T07:27:20.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:27:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:20.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:20 smithi064 ceph-mon[24760]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:20.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:20 smithi064 ceph-mon[24760]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T07:27:20.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:20 smithi064 ceph-mon[29477]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:20.742 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:20 smithi064 ceph-mon[29477]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T07:27:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:20 smithi136 ceph-mon[25575]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:20 smithi136 ceph-mon[25575]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T07:27:22.268 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:27:22.268 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":57,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:22:20.219139+0000","modified":"2024-02-10T07:27:21.687644+0000","last_up_change":"2024-02-10T07:27:18.666943+0000","last_in_change":"2024-02-10T07:27:02.125071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:25:23.149675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f5938cc8-fb10-459a-a41c-b568fa16a3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6803","nonce":1428254601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6805","nonce":1428254601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6809","nonce":1428254601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6807","nonce":1428254601}]},"public_addr":"172.21.15.64:6803/1428254601","cluster_addr":"172.21.15.64:6805/1428254601","heartbeat_back_addr":"172.21.15.64:6809/1428254601","heartbeat_front_addr":"172.21.15.64:6807/1428254601","state":["exists","up"]},{"osd":1,"uuid":"c5e6f297-b17d-4a61-98f6-851a3e10b0d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6811","nonce":3004829485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6813","nonce":3004829485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6817","nonce":3004829485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6815","nonce":3004829485}]},"public_addr":"172.21.15.64:6811/3004829485","cluster_addr":"172.21.15.64:6813/3004829485","heartbeat_back_addr":"172.21.15.64:6817/3004829485","heartbeat_front_addr":"172.21.15.64:6815/3004829485","state":["exists","up"]},{"osd":2,"uuid":"f5edca33-a497-4211-a1ab-1c0e448c2b2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6819","nonce":2748450094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6821","nonce":2748450094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6825","nonce":2748450094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6823","nonce":2748450094}]},"public_addr":"172.21.15.64:6819/2748450094","cluster_addr":"172.21.15.64:6821/2748450094","heartbeat_back_addr":"172.21.15.64:6825/2748450094","heartbeat_front_addr":"172.21.15.64:6823/2748450094","state":["exists","up"]},{"osd":3,"uuid":"e28605d6-1761-45da-813c-e41956e65e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6827","nonce":2011470016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6829","nonce":2011470016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6833","nonce":2011470016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6831","nonce":2011470016}]},"public_addr":"172.21.15.64:6827/2011470016","cluster_addr":"172.21.15.64:6829/2011470016","heartbeat_back_addr":"172.21.15.64:6833/2011470016","heartbeat_front_addr":"172.21.15.64:6831/2011470016","state":["exists","up"]},{"osd":4,"uuid":"b71b3cb7-13dc-400e-90a3-40212186dd8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6801","nonce":2071893520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6803","nonce":2071893520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6807","nonce":2071893520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6805","nonce":2071893520}]},"public_addr":"172.21.15.136:6801/2071893520","cluster_addr":"172.21.15.136:6803/2071893520","heartbeat_back_addr":"172.21.15.136:6807/2071893520","heartbeat_front_addr":"172.21.15.136:6805/2071893520","state":["exists","up"]},{"osd":5,"uuid":"a87215f6-b76a-4354-b969-8a6d6cb9c5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6809","nonce":1778300341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6811","nonce":1778300341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6815","nonce":1778300341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6813","nonce":1778300341}]},"public_addr":"172.21.15.136:6809/1778300341","cluster_addr":"172.21.15.136:6811/1778300341","heartbeat_back_addr":"172.21.15.136:6815/1778300341","heartbeat_front_addr":"172.21.15.136:6813/1778300341","state":["exists","up"]},{"osd":6,"uuid":"6cc79b12-52f6-409c-873d-ff5537c2e260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6817","nonce":3781698804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6819","nonce":3781698804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6823","nonce":3781698804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6821","nonce":3781698804}]},"public_addr":"172.21.15.136:6817/3781698804","cluster_addr":"172.21.15.136:6819/3781698804","heartbeat_back_addr":"172.21.15.136:6823/3781698804","heartbeat_front_addr":"172.21.15.136:6821/3781698804","state":["exists","up"]},{"osd":7,"uuid":"6369935e-1d08-4497-8930-19f4d84f7291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6825","nonce":4091613306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6827","nonce":4091613306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6831","nonce":4091613306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6829","nonce":4091613306}]},"public_addr":"172.21.15.136:6825/4091613306","cluster_addr":"172.21.15.136:6827/4091613306","heartbeat_back_addr":"172.21.15.136:6831/4091613306","heartbeat_front_addr":"172.21.15.136:6829/4091613306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:24:33.513484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:24:57.092358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:25:21.168013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:25:45.330097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:07.011390+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:30.015139+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:53.488935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:27:17.646534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/2854614764":"2024-02-11T07:23:12.773085+0000","172.21.15.64:6801/1756309217":"2024-02-11T07:23:12.773085+0000","172.21.15.64:6800/1756309217":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/2794690239":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/1586222082":"2024-02-11T07:22:50.214070+0000","172.21.15.64:0/3353043261":"2024-02-11T07:22:50.214070+0000","172.21.15.64:0/1410740193":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/184722407":"2024-02-11T07:22:50.214070+0000","172.21.15.64:6800/3057373934":"2024-02-11T07:22:50.214070+0000","172.21.15.64:6801/3057373934":"2024-02-11T07:22:50.214070+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:27:22.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:22.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: Detected new or changed devices on smithi136 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi136 to 4272M 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[24760]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: Detected new or changed devices on smithi136 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi136 to 4272M 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:22.270 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.271 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:22 smithi064 ceph-mon[29477]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T07:27:22.374 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:22.374 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: Detected new or changed devices on smithi136 2024-02-10T07:27:22.374 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.374 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.374 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi136 to 4272M 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:22.375 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:22 smithi136 ceph-mon[25575]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T07:27:22.951 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T07:25:23.149675+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-10T07:27:22.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T07:27:23.170 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:23.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:23 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/628220736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:27:23.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:23 smithi064 ceph-mon[24760]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T07:27:23.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:23 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/628220736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:27:23.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:23 smithi064 ceph-mon[29477]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T07:27:23.387 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:23 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/628220736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:27:23.387 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:23 smithi136 ceph-mon[25575]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T07:27:24.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:24 smithi136 ceph-mon[25575]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-10T07:27:25.041 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-02-10T07:27:25.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:24 smithi064 ceph-mon[24760]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-10T07:27:25.041 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:24 smithi064 ceph-mon[29477]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-10T07:27:25.741 INFO:tasks.cephadm:Adding prometheus.a on smithi136 2024-02-10T07:27:25.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi136=a' 2024-02-10T07:27:25.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:25 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3422657739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T07:27:26.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:25 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3422657739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T07:27:26.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:25 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3422657739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T07:27:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:26 smithi136 ceph-mon[25575]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:27:27.067 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:26 smithi064 ceph-mon[24760]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:27:27.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:26 smithi064 ceph-mon[29477]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:27:27.819 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled prometheus update... 2024-02-10T07:27:28.456 DEBUG:teuthology.orchestra.run.smithi136:prometheus.a> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T07:27:28.459 INFO:tasks.cephadm:Adding node-exporter.a on smithi064 2024-02-10T07:27:28.459 INFO:tasks.cephadm:Adding node-exporter.b on smithi136 2024-02-10T07:27:28.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi064=a;smithi136=b' 2024-02-10T07:27:29.055 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ignoring --setuser ceph since I am not root 2024-02-10T07:27:29.055 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ignoring --setgroup ceph since I am not root 2024-02-10T07:27:29.055 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mgr[26759]: -- 172.21.15.136:0/2910399964 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ea5955e340 con 0x55ea59536c00 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: Saving service prometheus spec with placement smithi136=a;count:1 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:29.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:28 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T07:27:29.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-02-10T07:27:29.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:29.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: Saving service prometheus spec with placement smithi136=a;count:1 2024-02-10T07:27:29.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:29.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setuser ceph since I am not root 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setgroup ceph since I am not root 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mgr[24998]: -- 172.21.15.64:0/1356703606 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557eed262340 con 0x557eed23b000 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-02-10T07:27:29.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: Saving service prometheus spec with placement smithi136=a;count:1 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' 2024-02-10T07:27:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:28 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T07:27:29.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:29.381+0000 7ff8b4389000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:27:29.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:29.472+0000 7ff8b4389000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:27:29.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:29 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:29.391+0000 7f96cdd80000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:27:29.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:29 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:29.482+0000 7f96cdd80000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:27:30.125 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:29 smithi136 ceph-mon[25575]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T07:27:30.125 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:29 smithi136 ceph-mon[25575]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T07:27:30.125 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.078+0000 7ff8b4389000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:27:30.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:29 smithi064 ceph-mon[24760]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T07:27:30.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:29 smithi064 ceph-mon[24760]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T07:27:30.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:29 smithi064 ceph-mon[29477]: from='mgr.14150 172.21.15.64:0/2952338267' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T07:27:30.182 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:29 smithi064 ceph-mon[29477]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T07:27:30.182 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.091+0000 7f96cdd80000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:27:30.182 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.181+0000 7f96cdd80000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:27:30.411 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.171+0000 7ff8b4389000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:27:30.412 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.255+0000 7ff8b4389000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:27:30.412 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.411+0000 7ff8b4389000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:27:30.499 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.266+0000 7f96cdd80000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:27:30.499 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.417+0000 7f96cdd80000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:27:30.499 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.499+0000 7f96cdd80000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:27:30.684 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.493+0000 7ff8b4389000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:27:30.684 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.602+0000 7ff8b4389000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:27:30.684 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.684+0000 7ff8b4389000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:27:30.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.614+0000 7f96cdd80000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:27:30.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.699+0000 7f96cdd80000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:27:30.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.846+0000 7ff8b4389000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:27:30.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:30.930+0000 7ff8b4389000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:27:31.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.865+0000 7f96cdd80000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:27:31.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:30 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:30.952+0000 7f96cdd80000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:27:32.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:31.943+0000 7ff8b4389000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:27:32.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:32.071+0000 7ff8b4389000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:27:32.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:31 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:31.998+0000 7f96cdd80000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:27:32.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:32.130+0000 7f96cdd80000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:27:32.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:32.258+0000 7ff8b4389000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:27:32.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:32.344+0000 7ff8b4389000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:27:32.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:32.477+0000 7ff8b4389000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:27:32.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:32.325+0000 7f96cdd80000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:27:32.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:32.414+0000 7f96cdd80000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:27:32.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:32.554+0000 7f96cdd80000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:27:33.094 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:32.795+0000 7ff8b4389000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:27:33.094 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:33.010+0000 7ff8b4389000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:27:33.094 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:33.093+0000 7ff8b4389000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:27:33.202 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:32.886+0000 7f96cdd80000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:27:33.202 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:33.114+0000 7f96cdd80000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:27:33.202 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:33.202+0000 7f96cdd80000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:27:33.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:33.260+0000 7ff8b4389000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:27:33.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:33.377+0000 7f96cdd80000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:27:34.447 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:34.161+0000 7ff8b4389000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:27:34.625 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:34.319+0000 7f96cdd80000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:27:34.700 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:34.446+0000 7ff8b4389000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:27:34.700 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:34.538+0000 7ff8b4389000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:27:34.700 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:34.683+0000 7ff8b4389000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:27:34.700 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: [10/Feb/2024:07:27:34] ENGINE Bus STARTING 2024-02-10T07:27:34.881 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:34.625+0000 7f96cdd80000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:27:34.881 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:34.725+0000 7f96cdd80000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:27:34.881 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:34.881+0000 7f96cdd80000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:27:34.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: CherryPy Checker: 2024-02-10T07:27:34.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: The Application mounted at '' has an empty config. 2024-02-10T07:27:34.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: 2024-02-10T07:27:34.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: [10/Feb/2024:07:27:34] ENGINE Serving on http://:::9283 2024-02-10T07:27:34.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:27:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: [10/Feb/2024:07:27:34] ENGINE Bus STARTED 2024-02-10T07:27:35.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: Standby manager daemon x restarted 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: Standby manager daemon x started 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: Active manager daemon y restarted 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: Activating manager daemon y 2024-02-10T07:27:35.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:35 smithi136 ceph-mon[25575]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T07:27:35.993 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:27:35] ENGINE Bus STARTING 2024-02-10T07:27:35.993 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: CherryPy Checker: 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: Standby manager daemon x restarted 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: Standby manager daemon x started 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: Active manager daemon y restarted 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: Activating manager daemon y 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[29477]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T07:27:35.994 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: Standby manager daemon x restarted 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: Standby manager daemon x started 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/435884565' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: Active manager daemon y restarted 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: Activating manager daemon y 2024-02-10T07:27:35.995 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-mon[24760]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T07:27:36.244 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: The Application mounted at '' has an empty config. 2024-02-10T07:27:36.244 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:27:36.244 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:27:35] ENGINE Serving on http://:::9283 2024-02-10T07:27:36.245 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:27:35] ENGINE Bus STARTED 2024-02-10T07:27:36.678 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled node-exporter update... 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: mgrmap e17: y(active, starting, since 0.759466s), standbys: x 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:27:36.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: Manager daemon y is now available 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:36.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:36.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:37.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:27:36] ENGINE Bus STARTING 2024-02-10T07:27:37.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:27:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:27:37.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:27:36] ENGINE Bus STARTED 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: mgrmap e17: y(active, starting, since 0.759466s), standbys: x 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:27:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: Manager daemon y is now available 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:37.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: mgrmap e17: y(active, starting, since 0.759466s), standbys: x 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:27:37.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: Manager daemon y is now available 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:37.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:37.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:37.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:37.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: mgrmap e18: y(active, since 1.76269s), standbys: x 2024-02-10T07:27:37.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: Saving service node-exporter spec with placement smithi064=a;smithi136=b;count:2 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: [10/Feb/2024:07:27:36] ENGINE Bus STARTING 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: [10/Feb/2024:07:27:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: [10/Feb/2024:07:27:36] ENGINE Bus STARTED 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.769 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:37 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.790 DEBUG:teuthology.orchestra.run.smithi064:node-exporter.a> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T07:27:37.792 DEBUG:teuthology.orchestra.run.smithi136:node-exporter.b> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T07:27:37.794 INFO:tasks.cephadm:Adding alertmanager.a on smithi064 2024-02-10T07:27:37.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi064=a' 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: mgrmap e18: y(active, since 1.76269s), standbys: x 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: Saving service node-exporter spec with placement smithi064=a;smithi136=b;count:2 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: [10/Feb/2024:07:27:36] ENGINE Bus STARTING 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: [10/Feb/2024:07:27:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:27:37.872 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: [10/Feb/2024:07:27:36] ENGINE Bus STARTED 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: mgrmap e18: y(active, since 1.76269s), standbys: x 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: Saving service node-exporter spec with placement smithi064=a;smithi136=b;count:2 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: [10/Feb/2024:07:27:36] ENGINE Bus STARTING 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: [10/Feb/2024:07:27:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: [10/Feb/2024:07:27:36] ENGINE Bus STARTED 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:37.873 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:37 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:38 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:38 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:39.994 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled alertmanager update... 2024-02-10T07:27:40.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:39 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:40.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:39 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:27:40.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:39 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:40.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:39 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:27:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:40.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:40.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:40.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:27:40.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:40.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:39 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:40.743 DEBUG:teuthology.orchestra.run.smithi064:alertmanager.a> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T07:27:40.745 INFO:tasks.cephadm:Adding grafana.a on smithi136 2024-02-10T07:27:40.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi136=a' 2024-02-10T07:27:40.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:40 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:40 smithi136 ceph-mon[25575]: Deploying daemon prometheus.a on smithi136 2024-02-10T07:27:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:40 smithi136 ceph-mon[25575]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:40 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[24760]: Deploying daemon prometheus.a on smithi136 2024-02-10T07:27:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[24760]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:41.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:41.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[29477]: Deploying daemon prometheus.a on smithi136 2024-02-10T07:27:41.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[29477]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:41.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:40 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:41 smithi136 ceph-mon[25575]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:41 smithi136 ceph-mon[25575]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-02-10T07:27:42.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:41 smithi064 ceph-mon[24760]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:42.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:41 smithi064 ceph-mon[24760]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-02-10T07:27:42.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:41 smithi064 ceph-mon[29477]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:42.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:41 smithi064 ceph-mon[29477]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-02-10T07:27:43.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:42 smithi064 ceph-mon[24760]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:43.216 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:42 smithi064 ceph-mon[29477]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:43.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:42 smithi136 ceph-mon[25575]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:43.867 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled grafana update... 2024-02-10T07:27:44.880 DEBUG:teuthology.orchestra.run.smithi136:grafana.a> sudo journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T07:27:44.882 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T07:27:44.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T07:27:45.195 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[24760]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:45.195 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[24760]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:45.195 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[24760]: Saving service grafana spec with placement smithi136=a;count:1 2024-02-10T07:27:45.195 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:45.195 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[29477]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:45.195 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[29477]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:45.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[29477]: Saving service grafana spec with placement smithi136=a;count:1 2024-02-10T07:27:45.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:44 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:45.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:44 smithi136 ceph-mon[25575]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:44 smithi136 ceph-mon[25575]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:44 smithi136 ceph-mon[25575]: Saving service grafana spec with placement smithi136=a;count:1 2024-02-10T07:27:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:44 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:46.928 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-02-10T07:27:46.929 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQByJcdl+E4LNxAAoG2JPECuz5ErdrQtl7w8DA== 2024-02-10T07:27:47.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:46 smithi136 ceph-mon[25575]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:47.239 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:46 smithi064 ceph-mon[24760]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:47.240 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:46 smithi064 ceph-mon[29477]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:47.547 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T07:27:47.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T07:27:47.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T07:27:47.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T07:27:48.192 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:47 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1899922502' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:48.192 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:47 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:48.192 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:47 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:27:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:47 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1899922502' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:47 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:47 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:27:48.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:47 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1899922502' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:48.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:47 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:48.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:47 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:27:49.183 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:48 smithi136 ceph-mon[25575]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:49.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:48 smithi064 ceph-mon[24760]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:49.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:48 smithi064 ceph-mon[29477]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:50 smithi136 ceph-mon[25575]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:50 smithi064 ceph-mon[24760]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:50 smithi064 ceph-mon[29477]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:51.171 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-02-10T07:27:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQB3JcdlmQD9CRAA+WMI9Se9ncw0dgZtxHGLlA== 2024-02-10T07:27:51.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:51 smithi136 ceph-mon[25575]: from='client.? 172.21.15.136:0/136941084' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:51.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:51 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:51.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:51 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:27:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:51 smithi064 ceph-mon[24760]: from='client.? 172.21.15.136:0/136941084' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:51 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:51 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:27:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:51 smithi064 ceph-mon[29477]: from='client.? 172.21.15.136:0/136941084' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:51 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:27:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:51 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:27:52.915 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:52 smithi136 ceph-mon[25575]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:52.917 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-02-10T07:27:52.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T07:27:52.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T07:27:52.962 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T07:27:52.963 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T07:27:52.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T07:27:52.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:52 smithi064 ceph-mon[24760]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:52.992 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:52 smithi064 ceph-mon[29477]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:53.189 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:54.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:54 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:54.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:54 smithi064 ceph-mon[24760]: Deploying daemon node-exporter.a on smithi064 2024-02-10T07:27:54.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:54 smithi064 ceph-mon[24760]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:54.575 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:54 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:54.575 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:54 smithi064 ceph-mon[29477]: Deploying daemon node-exporter.a on smithi064 2024-02-10T07:27:54.575 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:54 smithi064 ceph-mon[29477]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:54.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:54 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:54.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:54 smithi136 ceph-mon[25575]: Deploying daemon node-exporter.a on smithi064 2024-02-10T07:27:54.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:54 smithi136 ceph-mon[25575]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:55.193 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:27:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:55 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/132418077' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T07:27:55.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:55 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/132418077' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T07:27:55.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:55 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/132418077' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T07:27:56.058 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":19,"active_gid":14421,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1852510286},{"type":"v1","addr":"172.21.15.64:6801","nonce":1852510286}]},"active_addr":"172.21.15.64:6801/1852510286","active_change":"2024-02-10T07:27:34.894576+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34310,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/","prometheus":"http://172.21.15.64:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1546573538}]},{"addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":716243094}]},{"addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":4253215926}]},{"addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1442184411}]}]}} 2024-02-10T07:27:56.061 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T07:27:56.061 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T07:27:56.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:27:56.888 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:56.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:56 smithi136 ceph-mon[25575]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:56 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:56.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:56 smithi064 ceph-mon[24760]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:56.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:56 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:56.987 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:56 smithi064 ceph-mon[29477]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:56.988 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:56 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:57.814 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:57 smithi064 podman[44203]: 2024-02-10 07:27:57.696610517 +0000 UTC m=+2.659640341 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T07:27:58.189 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:57 smithi064 podman[44203]: 2024-02-10 07:27:57.903718753 +0000 UTC m=+2.866748605 container create 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 podman[44203]: 2024-02-10 07:27:58.114424203 +0000 UTC m=+3.077454025 container init 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 podman[44203]: 2024-02-10 07:27:58.118034617 +0000 UTC m=+3.081064436 container start 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.122Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.122Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.122Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.122Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T07:27:58.190 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T07:27:58.191 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T07:27:58.192 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T07:27:58.193 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.124Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T07:27:58.194 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[44359]: ts=2024-02-10T07:27:58.125Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T07:27:58.194 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 bash[44203]: 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 2024-02-10T07:27:58.564 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:27:58 smithi064 systemd[1]: Started Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:27:58.950 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:58 smithi136 ceph-mon[25575]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:58.950 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:58 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:58.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-mon[24760]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:58.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:58.962 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-mon[29477]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:27:58.962 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:58 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:27:59.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:27:59 smithi136 bash[40917]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-10T07:27:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:27:59.384 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":59,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:22:20.219139+0000","modified":"2024-02-10T07:27:34.894170+0000","last_up_change":"2024-02-10T07:27:18.666943+0000","last_in_change":"2024-02-10T07:27:02.125071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:25:23.149675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f5938cc8-fb10-459a-a41c-b568fa16a3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6803","nonce":1428254601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6805","nonce":1428254601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6809","nonce":1428254601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6807","nonce":1428254601}]},"public_addr":"172.21.15.64:6803/1428254601","cluster_addr":"172.21.15.64:6805/1428254601","heartbeat_back_addr":"172.21.15.64:6809/1428254601","heartbeat_front_addr":"172.21.15.64:6807/1428254601","state":["exists","up"]},{"osd":1,"uuid":"c5e6f297-b17d-4a61-98f6-851a3e10b0d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6811","nonce":3004829485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6813","nonce":3004829485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6817","nonce":3004829485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6815","nonce":3004829485}]},"public_addr":"172.21.15.64:6811/3004829485","cluster_addr":"172.21.15.64:6813/3004829485","heartbeat_back_addr":"172.21.15.64:6817/3004829485","heartbeat_front_addr":"172.21.15.64:6815/3004829485","state":["exists","up"]},{"osd":2,"uuid":"f5edca33-a497-4211-a1ab-1c0e448c2b2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6819","nonce":2748450094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6821","nonce":2748450094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6825","nonce":2748450094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6823","nonce":2748450094}]},"public_addr":"172.21.15.64:6819/2748450094","cluster_addr":"172.21.15.64:6821/2748450094","heartbeat_back_addr":"172.21.15.64:6825/2748450094","heartbeat_front_addr":"172.21.15.64:6823/2748450094","state":["exists","up"]},{"osd":3,"uuid":"e28605d6-1761-45da-813c-e41956e65e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6827","nonce":2011470016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6829","nonce":2011470016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6833","nonce":2011470016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6831","nonce":2011470016}]},"public_addr":"172.21.15.64:6827/2011470016","cluster_addr":"172.21.15.64:6829/2011470016","heartbeat_back_addr":"172.21.15.64:6833/2011470016","heartbeat_front_addr":"172.21.15.64:6831/2011470016","state":["exists","up"]},{"osd":4,"uuid":"b71b3cb7-13dc-400e-90a3-40212186dd8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6801","nonce":2071893520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6803","nonce":2071893520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6807","nonce":2071893520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6805","nonce":2071893520}]},"public_addr":"172.21.15.136:6801/2071893520","cluster_addr":"172.21.15.136:6803/2071893520","heartbeat_back_addr":"172.21.15.136:6807/2071893520","heartbeat_front_addr":"172.21.15.136:6805/2071893520","state":["exists","up"]},{"osd":5,"uuid":"a87215f6-b76a-4354-b969-8a6d6cb9c5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6809","nonce":1778300341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6811","nonce":1778300341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6815","nonce":1778300341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6813","nonce":1778300341}]},"public_addr":"172.21.15.136:6809/1778300341","cluster_addr":"172.21.15.136:6811/1778300341","heartbeat_back_addr":"172.21.15.136:6815/1778300341","heartbeat_front_addr":"172.21.15.136:6813/1778300341","state":["exists","up"]},{"osd":6,"uuid":"6cc79b12-52f6-409c-873d-ff5537c2e260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6817","nonce":3781698804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6819","nonce":3781698804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6823","nonce":3781698804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6821","nonce":3781698804}]},"public_addr":"172.21.15.136:6817/3781698804","cluster_addr":"172.21.15.136:6819/3781698804","heartbeat_back_addr":"172.21.15.136:6823/3781698804","heartbeat_front_addr":"172.21.15.136:6821/3781698804","state":["exists","up"]},{"osd":7,"uuid":"6369935e-1d08-4497-8930-19f4d84f7291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6825","nonce":4091613306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6827","nonce":4091613306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6831","nonce":4091613306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6829","nonce":4091613306}]},"public_addr":"172.21.15.136:6825/4091613306","cluster_addr":"172.21.15.136:6827/4091613306","heartbeat_back_addr":"172.21.15.136:6831/4091613306","heartbeat_front_addr":"172.21.15.136:6829/4091613306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:24:33.513484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:24:57.092358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:25:21.168013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:25:45.330097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:07.011390+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:30.015139+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:53.488935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:27:17.646534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/283510722":"2024-02-11T07:27:34.894114+0000","172.21.15.64:6801/2572554528":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/963865334":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/3390118538":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/2854614764":"2024-02-11T07:23:12.773085+0000","172.21.15.64:6801/1756309217":"2024-02-11T07:23:12.773085+0000","172.21.15.64:6800/1756309217":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/2794690239":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/2955412271":"2024-02-11T07:27:34.894114+0000","172.21.15.64:6800/2572554528":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/1586222082":"2024-02-11T07:22:50.214070+0000","172.21.15.64:0/3353043261":"2024-02-11T07:22:50.214070+0000","172.21.15.64:0/1410740193":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/184722407":"2024-02-11T07:22:50.214070+0000","172.21.15.64:6800/3057373934":"2024-02-11T07:22:50.214070+0000","172.21.15.64:6801/3057373934":"2024-02-11T07:22:50.214070+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:27:59.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:59 smithi136 ceph-mon[25575]: Deploying daemon node-exporter.b on smithi136 2024-02-10T07:27:59.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:27:59 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/529107442' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:27:59.973 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:27:59 smithi136 bash[40917]: Getting image source signatures 2024-02-10T07:27:59.973 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:27:59 smithi136 bash[40917]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-10T07:27:59.973 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:27:59 smithi136 bash[40917]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-10T07:27:59.973 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:27:59 smithi136 bash[40917]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-10T07:28:00.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:59 smithi064 ceph-mon[24760]: Deploying daemon node-exporter.b on smithi136 2024-02-10T07:28:00.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:27:59 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/529107442' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:28:00.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:59 smithi064 ceph-mon[29477]: Deploying daemon node-exporter.b on smithi136 2024-02-10T07:28:00.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:27:59 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/529107442' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:28:00.150 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T07:28:00.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:28:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:00.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:00 smithi064 ceph-mon[24760]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:00.802 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:00 smithi064 ceph-mon[29477]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:01.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:00 smithi136 ceph-mon[25575]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:01.150 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:00 smithi136 bash[40917]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-10T07:28:01.472 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 bash[40917]: Writing manifest to image destination 2024-02-10T07:28:01.793 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 podman[40917]: 2024-02-10 07:28:01.405647559 +0000 UTC m=+2.354650149 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T07:28:01.793 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 podman[40917]: 2024-02-10 07:28:01.550093761 +0000 UTC m=+2.499096397 container create bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:28:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:28:02.178 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":59,"fsid":"0ded8f14-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:22:20.219139+0000","modified":"2024-02-10T07:27:34.894170+0000","last_up_change":"2024-02-10T07:27:18.666943+0000","last_in_change":"2024-02-10T07:27:02.125071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:25:23.149675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f5938cc8-fb10-459a-a41c-b568fa16a3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6803","nonce":1428254601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6805","nonce":1428254601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6809","nonce":1428254601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1428254601},{"type":"v1","addr":"172.21.15.64:6807","nonce":1428254601}]},"public_addr":"172.21.15.64:6803/1428254601","cluster_addr":"172.21.15.64:6805/1428254601","heartbeat_back_addr":"172.21.15.64:6809/1428254601","heartbeat_front_addr":"172.21.15.64:6807/1428254601","state":["exists","up"]},{"osd":1,"uuid":"c5e6f297-b17d-4a61-98f6-851a3e10b0d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6811","nonce":3004829485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6813","nonce":3004829485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6817","nonce":3004829485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3004829485},{"type":"v1","addr":"172.21.15.64:6815","nonce":3004829485}]},"public_addr":"172.21.15.64:6811/3004829485","cluster_addr":"172.21.15.64:6813/3004829485","heartbeat_back_addr":"172.21.15.64:6817/3004829485","heartbeat_front_addr":"172.21.15.64:6815/3004829485","state":["exists","up"]},{"osd":2,"uuid":"f5edca33-a497-4211-a1ab-1c0e448c2b2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6819","nonce":2748450094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6821","nonce":2748450094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6825","nonce":2748450094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2748450094},{"type":"v1","addr":"172.21.15.64:6823","nonce":2748450094}]},"public_addr":"172.21.15.64:6819/2748450094","cluster_addr":"172.21.15.64:6821/2748450094","heartbeat_back_addr":"172.21.15.64:6825/2748450094","heartbeat_front_addr":"172.21.15.64:6823/2748450094","state":["exists","up"]},{"osd":3,"uuid":"e28605d6-1761-45da-813c-e41956e65e01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6827","nonce":2011470016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6829","nonce":2011470016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6833","nonce":2011470016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2011470016},{"type":"v1","addr":"172.21.15.64:6831","nonce":2011470016}]},"public_addr":"172.21.15.64:6827/2011470016","cluster_addr":"172.21.15.64:6829/2011470016","heartbeat_back_addr":"172.21.15.64:6833/2011470016","heartbeat_front_addr":"172.21.15.64:6831/2011470016","state":["exists","up"]},{"osd":4,"uuid":"b71b3cb7-13dc-400e-90a3-40212186dd8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6801","nonce":2071893520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6803","nonce":2071893520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6807","nonce":2071893520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2071893520},{"type":"v1","addr":"172.21.15.136:6805","nonce":2071893520}]},"public_addr":"172.21.15.136:6801/2071893520","cluster_addr":"172.21.15.136:6803/2071893520","heartbeat_back_addr":"172.21.15.136:6807/2071893520","heartbeat_front_addr":"172.21.15.136:6805/2071893520","state":["exists","up"]},{"osd":5,"uuid":"a87215f6-b76a-4354-b969-8a6d6cb9c5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6809","nonce":1778300341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6811","nonce":1778300341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6815","nonce":1778300341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1778300341},{"type":"v1","addr":"172.21.15.136:6813","nonce":1778300341}]},"public_addr":"172.21.15.136:6809/1778300341","cluster_addr":"172.21.15.136:6811/1778300341","heartbeat_back_addr":"172.21.15.136:6815/1778300341","heartbeat_front_addr":"172.21.15.136:6813/1778300341","state":["exists","up"]},{"osd":6,"uuid":"6cc79b12-52f6-409c-873d-ff5537c2e260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6817","nonce":3781698804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6819","nonce":3781698804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6823","nonce":3781698804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3781698804},{"type":"v1","addr":"172.21.15.136:6821","nonce":3781698804}]},"public_addr":"172.21.15.136:6817/3781698804","cluster_addr":"172.21.15.136:6819/3781698804","heartbeat_back_addr":"172.21.15.136:6823/3781698804","heartbeat_front_addr":"172.21.15.136:6821/3781698804","state":["exists","up"]},{"osd":7,"uuid":"6369935e-1d08-4497-8930-19f4d84f7291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6825","nonce":4091613306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6827","nonce":4091613306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6831","nonce":4091613306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4091613306},{"type":"v1","addr":"172.21.15.136:6829","nonce":4091613306}]},"public_addr":"172.21.15.136:6825/4091613306","cluster_addr":"172.21.15.136:6827/4091613306","heartbeat_back_addr":"172.21.15.136:6831/4091613306","heartbeat_front_addr":"172.21.15.136:6829/4091613306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:24:33.513484+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:24:57.092358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:25:21.168013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:25:45.330097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:07.011390+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:30.015139+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:26:53.488935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:27:17.646534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/283510722":"2024-02-11T07:27:34.894114+0000","172.21.15.64:6801/2572554528":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/963865334":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/3390118538":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/2854614764":"2024-02-11T07:23:12.773085+0000","172.21.15.64:6801/1756309217":"2024-02-11T07:23:12.773085+0000","172.21.15.64:6800/1756309217":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/2794690239":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/2955412271":"2024-02-11T07:27:34.894114+0000","172.21.15.64:6800/2572554528":"2024-02-11T07:27:34.894114+0000","172.21.15.64:0/1586222082":"2024-02-11T07:22:50.214070+0000","172.21.15.64:0/3353043261":"2024-02-11T07:22:50.214070+0000","172.21.15.64:0/1410740193":"2024-02-11T07:23:12.773085+0000","172.21.15.64:0/184722407":"2024-02-11T07:22:50.214070+0000","172.21.15.64:6800/3057373934":"2024-02-11T07:22:50.214070+0000","172.21.15.64:6801/3057373934":"2024-02-11T07:22:50.214070+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:28:02.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 podman[40917]: 2024-02-10 07:28:01.882090893 +0000 UTC m=+2.831093464 container init bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 podman[40917]: 2024-02-10 07:28:01.885222145 +0000 UTC m=+2.834224717 container start bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.889Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.889Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.889Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.889Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T07:28:02.223 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T07:28:02.224 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T07:28:02.225 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T07:28:02.226 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.890Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[40968]: ts=2024-02-10T07:28:01.891Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 bash[40917]: bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 2024-02-10T07:28:02.227 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:28:01 smithi136 systemd[1]: Started Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:28:02.827 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:02 smithi136 ceph-mon[25575]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:02.827 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:02 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:02.827 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:02 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:02.827 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:02 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:02.827 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:02 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:02.827 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:02 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4091543915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:28:02.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T07:28:02.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T07:28:02.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T07:28:02.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T07:28:02.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T07:28:02.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T07:28:02.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T07:28:02.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T07:28:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[24760]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4091543915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[29477]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:02 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4091543915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:28:03.495 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:03.505 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:03.768 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:03.768 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:03.768 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:04.777 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:04 smithi064 ceph-mon[24760]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:04.777 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:04 smithi064 ceph-mon[29477]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:05.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:04 smithi136 ceph-mon[25575]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:05.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:05 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:05.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:05 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:05.923 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:28:05 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:05] "GET /metrics HTTP/1.1" 200 191529 "" "Prometheus/2.33.4" 2024-02-10T07:28:05.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:05 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:05.923 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:05 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:06.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:05 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:06.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:05 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:07.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:06 smithi064 ceph-mon[24760]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:07.020 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:06 smithi064 ceph-mon[29477]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:07.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:06 smithi136 ceph-mon[25575]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:08.878 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:08 smithi064 ceph-mon[24760]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:08.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:08 smithi064 ceph-mon[29477]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:09.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:08 smithi136 ceph-mon[25575]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:09.854 INFO:teuthology.orchestra.run.smithi064.stdout:128849018906 2024-02-10T07:28:09.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T07:28:10.945 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:10 smithi064 ceph-mon[24760]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:10.945 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:10 smithi064 ceph-mon[29477]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:10 smithi136 ceph-mon[25575]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:11.303 INFO:teuthology.orchestra.run.smithi064.stdout:77309411363 2024-02-10T07:28:11.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T07:28:11.718 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:11.719 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:12.444 INFO:teuthology.orchestra.run.smithi064.stdout:34359738413 2024-02-10T07:28:12.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T07:28:12.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:28:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:28:12.726 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:12 smithi064 ceph-mon[24760]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:12.903 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:12.904 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:12.904 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:12.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:12 smithi136 ceph-mon[25575]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:13.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:12 smithi064 ceph-mon[29477]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:14.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:14 smithi064 ceph-mon[24760]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:14.987 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:14 smithi064 ceph-mon[29477]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:15.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:14 smithi136 ceph-mon[25575]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:15.752 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:28:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:15] "GET /metrics HTTP/1.1" 200 191516 "" "Prometheus/2.33.4" 2024-02-10T07:28:16.459 INFO:teuthology.orchestra.run.smithi064.stdout:163208757271 2024-02-10T07:28:16.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T07:28:17.057 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:17.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:16 smithi064 ceph-mon[29477]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:16 smithi064 ceph-mon[24760]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:17.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:16 smithi136 ceph-mon[25575]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:17.553 INFO:teuthology.orchestra.run.smithi064.stdout:55834574889 2024-02-10T07:28:17.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T07:28:18.400 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:18.406 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:19.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:18 smithi064 ceph-mon[24760]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:19.030 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:18 smithi064 ceph-mon[29477]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:19.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:18 smithi136 ceph-mon[25575]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:20 smithi136 ceph-mon[25575]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:20.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:20 smithi064 ceph-mon[24760]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:20.987 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:20 smithi064 ceph-mon[29477]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:22.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:28:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:28:22.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:22 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:22.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:22 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:22.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:22 smithi136 ceph-mon[25575]: Deploying daemon alertmanager.a on smithi064 2024-02-10T07:28:22.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:22 smithi136 ceph-mon[25575]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:22.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[24760]: Deploying daemon alertmanager.a on smithi064 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[24760]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[29477]: Deploying daemon alertmanager.a on smithi064 2024-02-10T07:28:22.559 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:22 smithi064 ceph-mon[29477]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:22.848 INFO:teuthology.orchestra.run.smithi064.stdout:128849018909 2024-02-10T07:28:23.184 INFO:teuthology.orchestra.run.smithi064.stdout:197568495635 2024-02-10T07:28:23.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T07:28:23.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:23 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1031069065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:28:23.498 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:23 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1031069065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:28:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:23 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1031069065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:28:24.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018906 got 128849018909 for osd.4 2024-02-10T07:28:24.195 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:24.452 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:24 smithi064 ceph-mon[24760]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:24.452 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:24 smithi064 ceph-mon[29477]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:24.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:24 smithi136 ceph-mon[25575]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:24.979 INFO:teuthology.orchestra.run.smithi064.stdout:107374182434 2024-02-10T07:28:24.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T07:28:25.417 INFO:teuthology.orchestra.run.smithi064.stdout:77309411366 2024-02-10T07:28:25.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:25 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2430650460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:28:25.744 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:28:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:25] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-02-10T07:28:25.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:25 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2430650460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:28:25.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:25 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2430650460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:28:26.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:26 smithi064 ceph-mon[24760]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:26.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:26 smithi064 ceph-mon[29477]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:26 smithi136 ceph-mon[25575]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:27.737 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:27.737 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:28.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:28 smithi064 ceph-mon[24760]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:28.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:28 smithi064 ceph-mon[29477]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:28.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:28 smithi136 ceph-mon[25575]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:29.510 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411366 for osd.2 2024-02-10T07:28:29.511 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:30.971 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:30 smithi064 ceph-mon[24760]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:30.972 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:30 smithi064 ceph-mon[29477]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:30.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:30 smithi136 ceph-mon[25575]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:31.347 INFO:teuthology.orchestra.run.smithi064.stdout:34359738417 2024-02-10T07:28:31.588 INFO:teuthology.orchestra.run.smithi064.stdout:55834574892 2024-02-10T07:28:31.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:31 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4151896867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:28:31.935 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:31 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/70907774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:28:31.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:31 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4151896867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:28:31.935 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:31 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/70907774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:28:31.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:31 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4151896867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:28:31.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:31 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/70907774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:28:32.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:28:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:28:32.542 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574892 for osd.1 2024-02-10T07:28:32.543 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:32.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:32 smithi064 ceph-mon[24760]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:32.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:32 smithi064 ceph-mon[29477]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:32.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738417 for osd.0 2024-02-10T07:28:32.953 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:32.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:32 smithi136 ceph-mon[25575]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:35.005 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:34 smithi064 ceph-mon[24760]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:35.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:34 smithi064 ceph-mon[29477]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:35.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:34 smithi136 ceph-mon[25575]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:35.929 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:28:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:35] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-02-10T07:28:36.005 INFO:teuthology.orchestra.run.smithi064.stdout:163208757275 2024-02-10T07:28:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[24760]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:28:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:28:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:28:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:28:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2217539330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:28:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[29477]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:28:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:28:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:28:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:28:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:36 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2217539330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:28:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:36 smithi136 ceph-mon[25575]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:28:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:28:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:28:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:28:37.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:36 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2217539330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:28:37.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757275 for osd.5 2024-02-10T07:28:37.802 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:39.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:38 smithi064 ceph-mon[24760]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:39.022 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:38 smithi064 ceph-mon[29477]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:39.102 INFO:teuthology.orchestra.run.smithi064.stdout:107374182437 2024-02-10T07:28:39.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:38 smithi136 ceph-mon[25575]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:39.537 INFO:teuthology.orchestra.run.smithi064.stdout:231928234001 2024-02-10T07:28:39.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T07:28:39.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:39 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3857722093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:28:39.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:39 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3857722093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:28:40.116 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:40.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:39 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3857722093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:28:40.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182437 for osd.3 2024-02-10T07:28:40.828 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:41.063 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:40 smithi064 ceph-mon[24760]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:41.063 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:40 smithi064 ceph-mon[29477]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:41.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:40 smithi136 ceph-mon[25575]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:42.062 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:41 smithi064 podman[47340]: 2024-02-10 07:28:41.88213786 +0000 UTC m=+0.349238659 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-10T07:28:42.062 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:42 smithi064 podman[47340]: 2024-02-10 07:28:42.061894994 +0000 UTC m=+0.528995783 container create 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:28:42.087 INFO:teuthology.orchestra.run.smithi064.stdout:197568495640 2024-02-10T07:28:42.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:28:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:28:42.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:42 smithi064 ceph-mon[24760]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:42.917 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:42 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3161727033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:28:42.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:42 smithi064 ceph-mon[29477]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:42.918 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:42 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3161727033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:28:42.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495635 got 197568495640 for osd.6 2024-02-10T07:28:42.942 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:43.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:42 smithi136 ceph-mon[25575]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:43.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:42 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3161727033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 podman[47340]: 2024-02-10 07:28:43.505067101 +0000 UTC m=+1.972167889 container init 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 podman[47340]: 2024-02-10 07:28:43.509029376 +0000 UTC m=+1.976130166 container start 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.521Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.521Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.523Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.64 port=9094 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.524Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.551Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.551Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.553Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:43.553Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T07:28:43.597 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 bash[47340]: 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed 2024-02-10T07:28:43.848 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:43 smithi064 systemd[1]: Started Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:28:44.253 INFO:teuthology.orchestra.run.smithi064.stdout:231928234003 2024-02-10T07:28:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:28:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: Deploying daemon grafana.a on smithi136 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1454257863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: Deploying daemon grafana.a on smithi136 2024-02-10T07:28:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:44 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1454257863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:28:44.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234001 got 231928234003 for osd.7 2024-02-10T07:28:44.935 DEBUG:teuthology.parallel:result is None 2024-02-10T07:28:44.936 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T07:28:44.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:28:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:28:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:28:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: Deploying daemon grafana.a on smithi136 2024-02-10T07:28:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:44 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1454257863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:28:45.160 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:45.540 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:45.525Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00090606s 2024-02-10T07:28:45.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:28:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:45] "GET /metrics HTTP/1.1" 200 191504 "" "Prometheus/2.33.4" 2024-02-10T07:28:46.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:46 smithi064 ceph-mon[24760]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:46.963 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:46 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:46.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:46 smithi064 ceph-mon[29477]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:46.964 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:46 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:28:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-02-10T07:28:47.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:46 smithi136 ceph-mon[25575]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:47.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:46 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:47.759 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-02-10T07:28:45.671933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48720,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912496,"statfs":{"total":767960285184,"available":767910395904,"internally_reserved":0,"allocated":6242304,"data_stored":3488556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002415"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":263,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-10T07:27:35.764212+0000","last_change":"2024-02-10T07:27:22.698794+0000","last_active":"2024-02-10T07:27:35.764212+0000","last_peered":"2024-02-10T07:27:35.764212+0000","last_clean":"2024-02-10T07:27:35.764212+0000","last_became_active":"2024-02-10T07:27:22.698292+0000","last_became_peered":"2024-02-10T07:27:22.698292+0000","last_unstale":"2024-02-10T07:27:35.764212+0000","last_undegraded":"2024-02-10T07:27:35.764212+0000","last_fullsized":"2024-02-10T07:27:35.764212+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:25:23.628116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:25:23.628116+0000","last_clean_scrub_stamp":"2024-02-10T07:25:23.628116+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:30:07.238469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.625}]},{"osd":1,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.64900000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.59999999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.59099999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.68500000000000005}]},{"osd":6,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55500000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.621},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.441}]},{"osd":1,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.504}]},{"osd":2,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.65900000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.69199999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.71999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.34599999999999997},"min":{"1min":0.26700000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.34100000000000003,"15min":0.32200000000000001},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.39800000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.36099999999999999},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.38},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56299999999999994,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.436},"min":{"1min":0.32600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.51900000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.59599999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46500000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.627,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.84499999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61699999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.69699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.34899999999999998,"15min":0.33000000000000002},"min":{"1min":0.23999999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.374,"15min":0.34799999999999998},"min":{"1min":0.22900000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.78200000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.39000000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.39800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68300000000000005,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.317}]},{"osd":3,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.96899999999999997,"5min":1.014,"15min":1.014},"last":0.432},{"interface":"front","average":{"1min":1.413,"5min":0.79300000000000004,"15min":0.68999999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":13.865,"5min":13.865,"15min":13.865},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.448,"5min":0.77900000000000003,"15min":0.66700000000000004},"min":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":13.804,"5min":13.804,"15min":13.804},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.57599999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.497}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6208,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738944,"statfs":{"total":95995035648,"available":95988678656,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.46300000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.247},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.29499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.46300000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.36199999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42199999999999999,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54000000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.48999999999999999,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46000000000000002,"15min":0.46200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.33400000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61899999999999999,"15min":0.63300000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.71299999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.70599999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75900000000000001,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.75},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68000000000000005}]},{"osd":7,"last update":"Sat Feb 10 07:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.60899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.34300000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59599999999999997,"5min":0.621,"15min":0.621},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.432},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.59399999999999997,"5min":0.623,"15min":0.623},"last":0.254},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47899999999999998,"15min":0.48099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.374},{"interface":"front","average":{"1min":0.44,"5min":0.47999999999999998,"15min":0.48699999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.28100000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.628,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.68100000000000005}]},{"osd":5,"last update":"Sat Feb 10 07:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66400000000000003,"15min":0.67600000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73599999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.621},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70299999999999996}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.434},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.496},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77500000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.754,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47499999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.27900000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.43099999999999999,"15min":0.41899999999999998},"min":{"1min":0.30299999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.502}]},{"osd":7,"last update":"Sat Feb 10 07:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.51400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5628,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739524,"statfs":{"total":95995035648,"available":95989272576,"internally_reserved":0,"allocated":577536,"data_stored":237473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55500000000000005}]},{"osd":1,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.502}]},{"osd":2,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.41599999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.435,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40699999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39300000000000002,"15min":0.38300000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.29199999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.36899999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.42099999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:28:47.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:28:47.980 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:49.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:48 smithi064 ceph-mon[24760]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:49.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:48 smithi064 ceph-mon[24760]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:49.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:48 smithi064 ceph-mon[29477]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:49.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:48 smithi064 ceph-mon[29477]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:49.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:48 smithi136 ceph-mon[25575]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:48 smithi136 ceph-mon[25575]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:28:49.768 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-02-10T07:28:50.316 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-02-10T07:28:49.672698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48720,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912496,"statfs":{"total":767960285184,"available":767910395904,"internally_reserved":0,"allocated":6242304,"data_stored":3488556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002350"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":263,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-10T07:27:35.764212+0000","last_change":"2024-02-10T07:27:22.698794+0000","last_active":"2024-02-10T07:27:35.764212+0000","last_peered":"2024-02-10T07:27:35.764212+0000","last_clean":"2024-02-10T07:27:35.764212+0000","last_became_active":"2024-02-10T07:27:22.698292+0000","last_became_peered":"2024-02-10T07:27:22.698292+0000","last_unstale":"2024-02-10T07:27:35.764212+0000","last_undegraded":"2024-02-10T07:27:35.764212+0000","last_fullsized":"2024-02-10T07:27:35.764212+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:25:23.628116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:25:23.628116+0000","last_clean_scrub_stamp":"2024-02-10T07:25:23.628116+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:30:07.238469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.501},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.63}]},{"osd":1,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.45300000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48599999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38500000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47299999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.437},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.41799999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Feb 10 07:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.626}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.34599999999999997},"min":{"1min":0.26700000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.34100000000000003,"15min":0.32200000000000001},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.36099999999999999},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56299999999999994,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.32600000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.436},"min":{"1min":0.32600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.51900000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.59599999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.32100000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.498}]},{"osd":5,"last update":"Sat Feb 10 07:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.627,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.47299999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Feb 10 07:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.34899999999999998,"15min":0.33000000000000002},"min":{"1min":0.23999999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.374,"15min":0.34799999999999998},"min":{"1min":0.22900000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.78200000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.39800000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68300000000000005,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.378}]},{"osd":3,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52400000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.96899999999999997,"5min":1.014,"15min":1.014},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.413,"5min":0.79300000000000004,"15min":0.68999999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":13.865,"5min":13.865,"15min":13.865},"last":0.69199999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.448,"5min":0.77900000000000003,"15min":0.66700000000000004},"min":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":13.804,"5min":13.804,"15min":13.804},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.622}]},{"osd":7,"last update":"Sat Feb 10 07:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6208,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738944,"statfs":{"total":95995035648,"available":95988678656,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.46300000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.29499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.46300000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42199999999999999,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.437},{"interface":"front","average":{"1min":0.371,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54000000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.48999999999999999,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46000000000000002,"15min":0.46200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61899999999999999,"15min":0.63300000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.71299999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.69399999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75900000000000001,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Feb 10 07:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.60099999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.34300000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59599999999999997,"5min":0.621,"15min":0.621},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.432},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.56699999999999995}]},{"osd":1,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.59399999999999997,"5min":0.623,"15min":0.623},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47899999999999998,"15min":0.48099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.42299999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.47999999999999998,"15min":0.48699999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.628,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66400000000000003,"15min":0.67600000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73599999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.64600000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77500000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.47599999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.754,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57499999999999996,"15min":0.58199999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.27900000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.374},{"interface":"front","average":{"1min":0.502,"5min":0.43099999999999999,"15min":0.41899999999999998},"min":{"1min":0.30299999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.503}]},{"osd":6,"last update":"Sat Feb 10 07:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.54700000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5628,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739524,"statfs":{"total":95995035648,"available":95989272576,"internally_reserved":0,"allocated":577536,"data_stored":237473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.505}]},{"osd":2,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.41599999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.435,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40699999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48699999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39300000000000002,"15min":0.38300000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.29199999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.45600000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.40100000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:28:50.319 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T07:28:50.320 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T07:28:50.320 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T07:28:50.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T07:28:50.539 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:50.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:50 smithi064 ceph-mon[24760]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:50.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:50 smithi064 ceph-mon[24760]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:50.970 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:50 smithi064 ceph-mon[29477]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:50.971 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:50 smithi064 ceph-mon[29477]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:50 smithi136 ceph-mon[25575]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:50 smithi136 ceph-mon[25575]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:52.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T07:28:52.328 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T07:28:52.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:28:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:28:52.923 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T07:28:52.924 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T07:28:52.924 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T07:28:52.933 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T07:28:52.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-02-10T07:28:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:52 smithi064 ceph-mon[24760]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:52 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2821065631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T07:28:53.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:52 smithi064 ceph-mon[29477]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:53.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:52 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2821065631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T07:28:53.078 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:52 smithi136 ceph-mon[25575]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:53.078 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:52 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2821065631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T07:28:53.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:28:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:28:53.527Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002995289s 2024-02-10T07:28:54.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:54 smithi136 ceph-mon[25575]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:54 smithi064 ceph-mon[24760]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:55.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:54 smithi064 ceph-mon[29477]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:55.587 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T07:28:55.596 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T07:28:55.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-02-10T07:28:55.783 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:28:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:28:55] "GET /metrics HTTP/1.1" 200 191511 "" "Prometheus/2.33.4" 2024-02-10T07:28:56.039 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:55 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' 2024-02-10T07:28:56.039 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:55 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' 2024-02-10T07:28:56.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:55 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' 2024-02-10T07:28:57.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:56 smithi064 ceph-mon[24760]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:57.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:56 smithi064 ceph-mon[29477]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:57.023 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:56 smithi136 ceph-mon[25575]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:57.023 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:56 smithi136 podman[41660]: 2024-02-10 07:28:56.88254543 +0000 UTC m=+0.017218915 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-10T07:28:57.023 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 podman[41660]: 2024-02-10 07:28:57.023009881 +0000 UTC m=+0.157683409 container create 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, architecture=x86_64, io.openshift.expose-services=, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, distribution-scope=public, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner ) 2024-02-10T07:28:57.290 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 podman[41660]: 2024-02-10 07:28:57.258879154 +0000 UTC m=+0.393552640 container init 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, name=ubi8, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git) 2024-02-10T07:28:57.290 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 podman[41660]: 2024-02-10 07:28:57.261587561 +0000 UTC m=+0.396261056 container start 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vendor=Red Hat, Inc., description=Ceph Grafana Container) 2024-02-10T07:28:57.540 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T07:28:57.541 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T07:28:57.542 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T07:28:57.542 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T07:28:57.542 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T07:28:57.542 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-10T07:28:57.542 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-10T07:28:57.542 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-10T07:28:57.543 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-10T07:28:57.544 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-10T07:28:57.545 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T07:28:57.546 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-10T07:28:57.547 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T07:28:57.548 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-10T07:28:57.549 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-10T07:28:57.550 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 bash[41660]: 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 2024-02-10T07:28:57.550 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 systemd[1]: Started Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-10T07:28:57.551 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-10T07:28:57.552 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T07:28:57.553 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-10T07:28:57.554 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T07:28:57.555 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-10T07:28:57.556 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-10T07:28:57.557 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-10T07:28:57.558 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T07:28:57.559 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T07:28:57.845 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-10T07:28:57.846 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-10T07:28:57.847 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-10T07:28:57.848 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-10T07:28:57.848 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-10T07:28:57.849 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-10T07:28:57.850 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-10T07:28:57.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-10T07:28:57.852 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-10T07:28:57.853 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-10T07:28:57.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-10T07:28:57.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-10T07:28:57.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-10T07:28:57.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-10T07:28:57.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T07:28:57.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-10T07:28:57.857 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T07:28:57.858 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T07:28:57.859 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-10T07:28:57.860 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-10T07:28:57.861 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-10T07:28:57.862 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-10T07:28:57.863 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-10T07:28:57.864 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=433.78241ms 2024-02-10T07:28:57.865 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-10T07:28:57.866 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:28:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:28:57+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T07:28:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T07:28:58.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2883729493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:28:58.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:28:58.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:58.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:58.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:58.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:58.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T07:28:58.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2883729493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:28:58.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:58 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:28:58.925 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:28:58.925 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "16dd835a-3272-4f17-99fb-3d022c71fae2", 2024-02-10T07:28:58.925 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "r", 2024-02-10T07:28:58.925 INFO:teuthology.orchestra.run.smithi064.stdout: "current_period": "6aa8c6b9-edb3-406f-a4f7-524fb2818571", 2024-02-10T07:28:58.926 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1 2024-02-10T07:28:58.926 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:28:58.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:28:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:28:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T07:28:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2883729493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:28:58.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:58 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:28:59.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:59 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T07:28:59.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:28:59 smithi136 ceph-mon[25575]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T07:29:00.284 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:59 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T07:29:00.284 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:28:59 smithi064 ceph-mon[29477]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T07:29:00.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:59 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T07:29:00.285 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:28:59 smithi064 ceph-mon[24760]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T07:29:00.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-02-10T07:29:01.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[24760]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:01.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[24760]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T07:29:01.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.030 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[29477]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:01.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[29477]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T07:29:01.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.031 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:00 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.126 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:00 smithi136 ceph-mon[25575]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:01.126 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:00 smithi136 ceph-mon[25575]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T07:29:01.126 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:00 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.126 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:00 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:01.126 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:00 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[24760]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:29:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[24760]: Reconfiguring daemon alertmanager.a on smithi064 2024-02-10T07:29:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[24760]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T07:29:02.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:02.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[29477]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:29:02.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[29477]: Reconfiguring daemon alertmanager.a on smithi064 2024-02-10T07:29:02.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:02 smithi064 ceph-mon[29477]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T07:29:02.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:29:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:29:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:02 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:02 smithi136 ceph-mon[25575]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:29:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:02 smithi136 ceph-mon[25575]: Reconfiguring daemon alertmanager.a on smithi064 2024-02-10T07:29:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:02 smithi136 ceph-mon[25575]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T07:29:02.474 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "ef6c81a8-254b-485e-845b-9df0dc0475ac", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "api_name": "default", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "is_master": "true", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "endpoints": [], 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "hostnames": [], 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "hostnames_s3website": [], 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "master_zone": "", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "zones": [], 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "placement_targets": [], 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "default_placement": "", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "realm_id": "16dd835a-3272-4f17-99fb-3d022c71fae2", 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_policy": { 2024-02-10T07:29:02.475 INFO:teuthology.orchestra.run.smithi064.stdout: "groups": [] 2024-02-10T07:29:02.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:02.476 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:29:03.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-02-10T07:29:04.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:04 smithi064 ceph-mon[24760]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T07:29:04.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:04 smithi064 ceph-mon[29477]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T07:29:05.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:04 smithi136 ceph-mon[25575]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-10T07:29:05.990 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:29:05 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:05] "GET /metrics HTTP/1.1" 200 191511 "" "Prometheus/2.33.4" 2024-02-10T07:29:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:06 smithi064 ceph-mon[29477]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-02-10T07:29:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:06 smithi064 ceph-mon[24760]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-02-10T07:29:07.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:06 smithi136 ceph-mon[25575]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-02-10T07:29:07.442 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:07 smithi064 systemd[1]: Stopping Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "edec88c7-7cf5-46a7-b23f-428fb6373408", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "z", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "domain_root": "z.rgw.meta:root", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "control_pool": "z.rgw.control", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "gc_pool": "z.rgw.log:gc", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "lc_pool": "z.rgw.log:lc", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "log_pool": "z.rgw.log", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "roles_pool": "z.rgw.meta:roles", 2024-02-10T07:29:07.446 INFO:teuthology.orchestra.run.smithi064.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "otp_pool": "z.rgw.otp", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "system_key": { 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "access_key": "", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "secret_key": "" 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "placement_pools": [ 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "key": "default-placement", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "val": { 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "index_pool": "z.rgw.buckets.index", 2024-02-10T07:29:07.447 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_classes": { 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: "STANDARD": { 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: "data_pool": "z.rgw.buckets.data" 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: "index_type": 0 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: "realm_id": "16dd835a-3272-4f17-99fb-3d022c71fae2", 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: "notif_pool": "z.rgw.log:notif" 2024-02-10T07:29:07.448 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:29:08.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-02-10T07:29:08.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:08 smithi064 bash[49077]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T07:29:08.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[47392]: level=info ts=2024-02-10T07:29:08.228Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:29:08.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:08 smithi064 podman[49189]: 2024-02-10 07:29:08.249060531 +0000 UTC m=+0.117658370 container died 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:08.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:08 smithi064 ceph-mon[24760]: pgmap v52: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 466 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-02-10T07:29:08.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:08 smithi064 ceph-mon[29477]: pgmap v52: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 466 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-02-10T07:29:08.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:08 smithi064 podman[49189]: 2024-02-10 07:29:08.628614928 +0000 UTC m=+0.497212765 container cleanup 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:08.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:08 smithi064 bash[49189]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T07:29:08.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:08 smithi064 bash[49240]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T07:29:08.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:08 smithi136 ceph-mon[25575]: pgmap v52: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 466 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-02-10T07:29:09.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:09 smithi064 podman[49210]: 2024-02-10 07:29:09.517258034 +0000 UTC m=+1.267835661 container remove 06b0d23c7996297915019aab2b4257795bed48372560e55bcdb324a135eb96ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:09.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:09 smithi064 podman[49210]: 2024-02-10 07:29:09.601214538 +0000 UTC m=+1.351792175 volume remove d4c2fe680a9b6cd7b9f25aaccb569328d681b8be822f98855c2eada11da72d7f 2024-02-10T07:29:09.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:09 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T07:29:09.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:09 smithi064 systemd[1]: Stopped Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:29:09.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:09 smithi064 systemd[1]: Starting Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:29:10.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:10 smithi064 ceph-mon[24760]: pgmap v53: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-02-10T07:29:10.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:10 smithi064 ceph-mon[29477]: pgmap v53: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-02-10T07:29:10.941 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:10 smithi064 podman[49404]: 2024-02-10 07:29:10.641040295 +0000 UTC m=+0.110034847 volume create 63f792d578e4da5efe215ff26d46dc8c3c208c2577428e2db8b84852305e7609 2024-02-10T07:29:10.941 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:10 smithi064 podman[49404]: 2024-02-10 07:29:10.549470669 +0000 UTC m=+0.018465234 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-10T07:29:10.941 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:10 smithi064 podman[49404]: 2024-02-10 07:29:10.758279007 +0000 UTC m=+0.227273572 container create 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:11.201 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:10 smithi136 ceph-mon[25575]: pgmap v53: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-02-10T07:29:11.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 podman[49404]: 2024-02-10 07:29:11.03573412 +0000 UTC m=+0.504728673 container init 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:11.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 podman[49404]: 2024-02-10 07:29:11.038204771 +0000 UTC m=+0.507199327 container start 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:11.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.048Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T07:29:11.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.048Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T07:29:11.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.050Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.64 port=9094 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.051Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.075Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.076Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.078Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:11.078Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 bash[49404]: 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 2024-02-10T07:29:11.315 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:11 smithi064 systemd[1]: Started Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:29:11.819 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:11 smithi136 ceph-mon[25575]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T07:29:11.819 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:11 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1553766256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:29:11.819 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:11 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:29:11.819 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:11 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:12.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:29:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:29:12.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[24760]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T07:29:12.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1553766256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:29:12.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:29:12.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:12.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[29477]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T07:29:12.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1553766256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:29:12.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:29:12.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:11 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:13.092 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:12 smithi136 ceph-mon[25575]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:29:13.092 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:12 smithi136 ceph-mon[25575]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:29:13.092 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:12 smithi136 ceph-mon[25575]: pgmap v55: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-10T07:29:13.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:12 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T07:29:13.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:12 smithi136 ceph-mon[25575]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T07:29:13.215 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:13.051Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000197625s 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[24760]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[24760]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[24760]: pgmap v55: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[24760]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[29477]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:29:13.216 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[29477]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:29:13.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[29477]: pgmap v55: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-10T07:29:13.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-10T07:29:13.217 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:12 smithi064 ceph-mon[29477]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T07:29:14.096 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:13 smithi136 ceph-mon[25575]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T07:29:14.096 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:13 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1553766256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:29:14.096 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:13 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:29:14.229 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-mon[24760]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T07:29:14.229 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1553766256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:29:14.229 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:29:14.229 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-mon[29477]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T07:29:14.229 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1553766256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:29:14.229 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:13 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:29:14.370 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 bash[42343]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:29:14.663 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:29:14.663 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T07:29:14.663 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T07:29:14.663 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T07:29:14.663 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T07:29:14.663 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.370Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[40674]: ts=2024-02-10T07:29:14.371Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T07:29:14.664 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 podman[42355]: 2024-02-10 07:29:14.380415466 +0000 UTC m=+0.104223346 container died 16039e19165cc00f7f93a3396f698f34a39e20bc82bc8b34dde50cbd48745357 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:14.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-mon[25575]: pgmap v58: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-02-10T07:29:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T07:29:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-mon[25575]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T07:29:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:14 smithi136 ceph-mon[25575]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T07:29:14.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 podman[42355]: 2024-02-10 07:29:14.761813178 +0000 UTC m=+0.485621057 container cleanup 16039e19165cc00f7f93a3396f698f34a39e20bc82bc8b34dde50cbd48745357 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:14.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 bash[42355]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T07:29:14.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:14 smithi136 bash[42381]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:29:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[24760]: pgmap v58: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-02-10T07:29:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T07:29:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[24760]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T07:29:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[24760]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T07:29:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[29477]: pgmap v58: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-02-10T07:29:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-10T07:29:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[29477]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T07:29:15.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:14 smithi064 ceph-mon[29477]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T07:29:15.289 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 podman[42369]: 2024-02-10 07:29:15.021522661 +0000 UTC m=+0.640104095 container remove 16039e19165cc00f7f93a3396f698f34a39e20bc82bc8b34dde50cbd48745357 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:15.289 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T07:29:15.289 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 systemd[1]: Stopped Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:29:15.289 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 systemd[1]: Starting Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:29:15.289 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 podman[42451]: 2024-02-10 07:29:15.159567844 +0000 UTC m=+0.018632588 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T07:29:15.289 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 podman[42451]: 2024-02-10 07:29:15.28951453 +0000 UTC m=+0.148579273 container create 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 podman[42451]: 2024-02-10 07:29:15.50017778 +0000 UTC m=+0.359242527 container init 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 podman[42451]: 2024-02-10 07:29:15.502826936 +0000 UTC m=+0.361891684 container start 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.526Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.526Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.526Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.526Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi136 (none))" 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.526Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T07:29:15.558 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.526Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.528Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.529Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.530Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.532Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.532Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.383µs 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.532Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T07:29:15.559 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 bash[42451]: 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 systemd[1]: Started Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.728Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.728Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.728Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.473µs wal_replay_duration=196.667833ms total_replay_duration=196.704326ms 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.730Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.730Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T07:29:15.849 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.730Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T07:29:15.850 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.746Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.611403ms db_storage=786ns remote_storage=1.305µs web_handler=338ns query_engine=561ns scrape=364.632µs scrape_sd=23.639µs notify=28.299µs notify_sd=7.023µs rules=14.967147ms 2024-02-10T07:29:15.850 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:29:15.746Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T07:29:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.64:9093"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.136:9095"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.136:3000"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:16.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[24760]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.64:9093"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.136:9095"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:29:16.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.136:3000"}]: dispatch 2024-02-10T07:29:16.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:16.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:16.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:29:16.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:15 smithi064 ceph-mon[29477]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T07:29:16.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:29:16.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:29:16.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.64:9093"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.136:9095"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.136:3000"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:29:16.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:15 smithi136 ceph-mon[25575]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T07:29:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:29:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.64:9093"}]: dispatch 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.136:9095"}]: dispatch 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.136:3000"}]: dispatch 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: pgmap v61: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:29:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:16 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:29:17.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:29:17.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.64:9093"}]: dispatch 2024-02-10T07:29:17.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:29:17.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.136:9095"}]: dispatch 2024-02-10T07:29:17.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:29:17.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.136:3000"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: pgmap v61: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.64:9093"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.136:9095"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:29:17.237 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.136:3000"}]: dispatch 2024-02-10T07:29:17.238 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: pgmap v61: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:17.238 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:29:17.238 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:16 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:29:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "e8744a69-b1cd-401d-a9ad-856ce6c1c5bb", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "predecessor_uuid": "6aa8c6b9-edb3-406f-a4f7-524fb2818571", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_status": [], 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "period_map": { 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "e8744a69-b1cd-401d-a9ad-856ce6c1c5bb", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "zonegroups": [ 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "ef6c81a8-254b-485e-845b-9df0dc0475ac", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "api_name": "default", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "is_master": "true", 2024-02-10T07:29:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "endpoints": [], 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "hostnames": [], 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "hostnames_s3website": [], 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "master_zone": "edec88c7-7cf5-46a7-b23f-428fb6373408", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "zones": [ 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "id": "edec88c7-7cf5-46a7-b23f-428fb6373408", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "z", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "endpoints": [], 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "log_meta": "false", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "log_data": "false", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "bucket_index_max_shards": 11, 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "read_only": "false", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_type": "", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_from_all": "true", 2024-02-10T07:29:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_from": [], 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_zone": "" 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "placement_targets": [ 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default-placement", 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_classes": [ 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "STANDARD" 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "default_placement": "default-placement", 2024-02-10T07:29:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "realm_id": "16dd835a-3272-4f17-99fb-3d022c71fae2", 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_policy": { 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "groups": [] 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "short_zone_ids": [ 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "key": "edec88c7-7cf5-46a7-b23f-428fb6373408", 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 3370636166 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "master_zonegroup": "ef6c81a8-254b-485e-845b-9df0dc0475ac", 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "master_zone": "edec88c7-7cf5-46a7-b23f-428fb6373408", 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "period_config": { 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "bucket_quota": { 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "user_quota": { 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2024-02-10T07:29:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "user_ratelimit": { 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_read_ops": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_write_ops": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_read_bytes": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_write_bytes": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "bucket_ratelimit": { 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_read_ops": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_write_ops": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_read_bytes": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max_write_bytes": 0, 2024-02-10T07:29:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "anonymous_ratelimit": { 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "max_read_ops": 0, 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "max_write_ops": 0, 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "max_read_bytes": 0, 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "max_write_bytes": 0, 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "realm_id": "16dd835a-3272-4f17-99fb-3d022c71fae2", 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "realm_name": "r", 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "realm_epoch": 2 2024-02-10T07:29:17.916 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:29:18.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:29:18.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[29477]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T07:29:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T07:29:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T07:29:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T07:29:18.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[29477]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T07:29:18.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:29:18.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[24760]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T07:29:18.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T07:29:18.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T07:29:18.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T07:29:18.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:17 smithi064 ceph-mon[24760]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T07:29:18.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:17 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:29:18.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:17 smithi136 ceph-mon[25575]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T07:29:18.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:17 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4226701942' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T07:29:18.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:17 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-10T07:29:18.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:17 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-10T07:29:18.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:17 smithi136 ceph-mon[25575]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T07:29:18.864 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:18 smithi136 ceph-mon[25575]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T07:29:18.865 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:18 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:19.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-02-10T07:29:19.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:18 smithi064 ceph-mon[29477]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T07:29:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:18 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:19.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:18 smithi064 ceph-mon[24760]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T07:29:19.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:18 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[24760]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 211 B/s rd, 422 B/s wr, 1 op/s 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:20.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:20 smithi064 ceph-mon[29477]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 211 B/s rd, 422 B/s wr, 1 op/s 2024-02-10T07:29:20.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:20 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:20 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:20.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:20 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:20.421 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:20 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:20.421 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:20 smithi136 ceph-mon[25575]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 211 B/s rd, 422 B/s wr, 1 op/s 2024-02-10T07:29:20.961 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled rgw.foo update... 2024-02-10T07:29:21.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:29:21.055Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004313431s 2024-02-10T07:29:21.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch apply rgw smpl' 2024-02-10T07:29:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:21 smithi136 ceph-mon[25575]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:21 smithi136 ceph-mon[25575]: Saving service rgw.foo spec with placement count:2 2024-02-10T07:29:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:21 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:21.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:21 smithi136 ceph-mon[25575]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T07:29:22.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[24760]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:22.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[24760]: Saving service rgw.foo spec with placement count:2 2024-02-10T07:29:22.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:22.244 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[24760]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T07:29:22.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[29477]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:22.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[29477]: Saving service rgw.foo spec with placement count:2 2024-02-10T07:29:22.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:22.245 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:21 smithi064 ceph-mon[29477]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T07:29:22.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:29:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:29:23.671 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled rgw.smpl update... 2024-02-10T07:29:24.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph osd pool create foo' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: Saving service rgw.smpl spec with placement count:2 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:24.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: Saving service rgw.smpl spec with placement count:2 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:24.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:24.943 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.943 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:24 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: Saving service rgw.smpl spec with placement count:2 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:24.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:24 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:25.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:25 smithi064 ceph-mon[24760]: Saving service rgw.foo spec with placement count:2 2024-02-10T07:29:25.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:25 smithi064 ceph-mon[24760]: Deploying daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:29:25.734 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:29:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:25] "GET /metrics HTTP/1.1" 200 196862 "" "Prometheus/2.33.4" 2024-02-10T07:29:25.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:25 smithi064 ceph-mon[29477]: Saving service rgw.foo spec with placement count:2 2024-02-10T07:29:25.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:25 smithi064 ceph-mon[29477]: Deploying daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:29:25.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:25 smithi136 ceph-mon[25575]: Saving service rgw.foo spec with placement count:2 2024-02-10T07:29:25.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:25 smithi136 ceph-mon[25575]: Deploying daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:29:26.696 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'foo' created 2024-02-10T07:29:26.885 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:26 smithi064 ceph-mon[24760]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 811 B/s wr, 2 op/s 2024-02-10T07:29:26.886 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:26 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3248678742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T07:29:26.886 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:26 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T07:29:26.886 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:26 smithi064 ceph-mon[29477]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 811 B/s wr, 2 op/s 2024-02-10T07:29:26.886 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:26 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3248678742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T07:29:26.886 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:26 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T07:29:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:26 smithi136 ceph-mon[25575]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 811 B/s wr, 2 op/s 2024-02-10T07:29:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:26 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3248678742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T07:29:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:26 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-10T07:29:27.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:27 smithi064 ceph-mon[24760]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-10T07:29:27.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:27 smithi064 ceph-mon[24760]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T07:29:27.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:27 smithi064 ceph-mon[29477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-10T07:29:27.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:27 smithi064 ceph-mon[29477]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T07:29:27.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:27 smithi136 ceph-mon[25575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-10T07:29:27.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:27 smithi136 ceph-mon[25575]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T07:29:28.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'rbd pool init foo' 2024-02-10T07:29:28.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:28 smithi064 ceph-mon[24760]: pgmap v71: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 729 B/s wr, 1 op/s 2024-02-10T07:29:28.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:28 smithi064 ceph-mon[24760]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T07:29:28.864 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:28 smithi064 ceph-mon[29477]: pgmap v71: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 729 B/s wr, 1 op/s 2024-02-10T07:29:28.864 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:28 smithi064 ceph-mon[29477]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T07:29:28.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:28 smithi136 ceph-mon[25575]: pgmap v71: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 729 B/s wr, 1 op/s 2024-02-10T07:29:28.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:28 smithi136 ceph-mon[25575]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T07:29:29.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T07:29:29.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:29.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:29.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:29.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:29.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:29.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:29 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T07:29:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:29.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:29.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:29.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:29.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:29.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:29 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:30.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:30 smithi136 ceph-mon[25575]: Deploying daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:29:30.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:30 smithi136 ceph-mon[25575]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:30.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:30 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/275021065' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T07:29:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:30 smithi064 ceph-mon[29477]: Deploying daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:29:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:30 smithi064 ceph-mon[29477]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:30 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/275021065' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T07:29:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:30 smithi064 ceph-mon[24760]: Deploying daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:29:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:30 smithi064 ceph-mon[24760]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:31.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:30 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/275021065' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-10T07:29:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/275021065' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-10T07:29:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T07:29:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/275021065' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:31.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:31 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/275021065' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:31.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:32.407 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:29:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:29:32.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:32 smithi136 ceph-mon[25575]: Saving service rgw.smpl spec with placement count:2 2024-02-10T07:29:32.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:32 smithi136 ceph-mon[25575]: Deploying daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:29:32.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:32 smithi136 ceph-mon[25575]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 222 KiB/s rd, 6.8 KiB/s wr, 424 op/s 2024-02-10T07:29:32.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:32 smithi136 ceph-mon[25575]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T07:29:33.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[24760]: Saving service rgw.smpl spec with placement count:2 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[24760]: Deploying daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[24760]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 222 KiB/s rd, 6.8 KiB/s wr, 424 op/s 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[24760]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[29477]: Saving service rgw.smpl spec with placement count:2 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[29477]: Deploying daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[29477]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 222 KiB/s rd, 6.8 KiB/s wr, 424 op/s 2024-02-10T07:29:33.054 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:32 smithi064 ceph-mon[29477]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T07:29:33.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch apply iscsi foo u p' 2024-02-10T07:29:34.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:33 smithi064 ceph-mon[24760]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T07:29:34.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:33 smithi064 ceph-mon[29477]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T07:29:34.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:33 smithi136 ceph-mon[25575]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T07:29:35.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 223 KiB/s rd, 6.8 KiB/s wr, 425 op/s 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 223 KiB/s rd, 6.8 KiB/s wr, 425 op/s 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:35.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:35.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:35.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:34 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 223 KiB/s rd, 6.8 KiB/s wr, 425 op/s 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:35.069 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:34 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:35.772 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:35] "GET /metrics HTTP/1.1" 200 196862 "" "Prometheus/2.33.4" 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: Deploying daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:29:35.773 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: Deploying daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:29:35.774 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:29:35.848 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: Deploying daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:29:35.849 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:29:36.182 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled iscsi.foo update... 2024-02-10T07:29:36.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 120' 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 5.7 KiB/s wr, 353 op/s 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:29:36.953 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:29:36.954 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:36 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 5.7 KiB/s wr, 353 op/s 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:29:37.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:37.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 5.7 KiB/s wr, 353 op/s 2024-02-10T07:29:37.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-10T07:29:37.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-10T07:29:37.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-10T07:29:37.007 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-02-10T07:29:37.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T07:29:37.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:37.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:29:37.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:29:37.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:36 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:38.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[29477]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:38.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[29477]: Saving service iscsi.foo spec with placement count:1 2024-02-10T07:29:38.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[29477]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T07:29:38.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:38.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:38.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:38.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[24760]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:38.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[24760]: Saving service iscsi.foo spec with placement count:1 2024-02-10T07:29:38.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[24760]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T07:29:38.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:38.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:38.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:37 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:37 smithi136 ceph-mon[25575]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:37 smithi136 ceph-mon[25575]: Saving service iscsi.foo spec with placement count:1 2024-02-10T07:29:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:37 smithi136 ceph-mon[25575]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T07:29:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:37 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:37 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:37 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:38.966 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:38 smithi064 ceph-mon[24760]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 515 B/s wr, 168 op/s 2024-02-10T07:29:38.967 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:38 smithi064 ceph-mon[29477]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 515 B/s wr, 168 op/s 2024-02-10T07:29:39.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:38 smithi136 ceph-mon[25575]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 515 B/s wr, 168 op/s 2024-02-10T07:29:40.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:40 smithi136 ceph-mon[25575]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T07:29:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:40 smithi136 ceph-mon[25575]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-02-10T07:29:40.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:40 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:40 smithi064 ceph-mon[24760]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T07:29:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:40 smithi064 ceph-mon[24760]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-02-10T07:29:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:40 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:40 smithi064 ceph-mon[29477]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T07:29:41.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:40 smithi064 ceph-mon[29477]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-02-10T07:29:41.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:40 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T07:29:41.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:41 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:42.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T07:29:42.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:41 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[29477]: Checking dashboard <-> RGW credentials 2024-02-10T07:29:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[29477]: Checking pool "foo" exists for service iscsi.foo 2024-02-10T07:29:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[29477]: Deploying daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:29:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[29477]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 223 KiB/s rd, 1023 B/s wr, 342 op/s 2024-02-10T07:29:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[29477]: mgrmap e20: y(active, since 2m), standbys: x 2024-02-10T07:29:43.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[24760]: Checking dashboard <-> RGW credentials 2024-02-10T07:29:43.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[24760]: Checking pool "foo" exists for service iscsi.foo 2024-02-10T07:29:43.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[24760]: Deploying daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:29:43.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[24760]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 223 KiB/s rd, 1023 B/s wr, 342 op/s 2024-02-10T07:29:43.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:42 smithi064 ceph-mon[24760]: mgrmap e20: y(active, since 2m), standbys: x 2024-02-10T07:29:43.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:42 smithi136 ceph-mon[25575]: Checking dashboard <-> RGW credentials 2024-02-10T07:29:43.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:42 smithi136 ceph-mon[25575]: Checking pool "foo" exists for service iscsi.foo 2024-02-10T07:29:43.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:42 smithi136 ceph-mon[25575]: Deploying daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:29:43.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:42 smithi136 ceph-mon[25575]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 223 KiB/s rd, 1023 B/s wr, 342 op/s 2024-02-10T07:29:43.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:42 smithi136 ceph-mon[25575]: mgrmap e20: y(active, since 2m), standbys: x 2024-02-10T07:29:44.777 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:44 smithi136 ceph-mon[25575]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 771 B/s wr, 257 op/s 2024-02-10T07:29:44.777 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:44 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:44.777 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:44 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:44.777 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:44 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[24760]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 771 B/s wr, 257 op/s 2024-02-10T07:29:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[29477]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 771 B/s wr, 257 op/s 2024-02-10T07:29:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:44 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:45.780 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:45] "GET /metrics HTTP/1.1" 200 209874 "" "Prometheus/2.33.4" 2024-02-10T07:29:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2116021742' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:29:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4074175570' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/283510722"}]: dispatch 2024-02-10T07:29:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:46.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2116021742' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:29:46.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4074175570' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/283510722"}]: dispatch 2024-02-10T07:29:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:45 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:46.107 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:45 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2116021742' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:29:46.107 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:45 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4074175570' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/283510722"}]: dispatch 2024-02-10T07:29:46.108 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:45 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:46 smithi136 ceph-mon[25575]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 342 B/s wr, 116 op/s 2024-02-10T07:29:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:46 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4074175570' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/283510722"}]': finished 2024-02-10T07:29:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:46 smithi136 ceph-mon[25575]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T07:29:46.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:46 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/96497648' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2572554528"}]: dispatch 2024-02-10T07:29:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[24760]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 342 B/s wr, 116 op/s 2024-02-10T07:29:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4074175570' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/283510722"}]': finished 2024-02-10T07:29:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[24760]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T07:29:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/96497648' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2572554528"}]: dispatch 2024-02-10T07:29:47.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[29477]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 342 B/s wr, 116 op/s 2024-02-10T07:29:47.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4074175570' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/283510722"}]': finished 2024-02-10T07:29:47.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[29477]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T07:29:47.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:46 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/96497648' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2572554528"}]: dispatch 2024-02-10T07:29:48.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/96497648' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2572554528"}]': finished 2024-02-10T07:29:48.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[24760]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T07:29:48.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2404291763' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]: dispatch 2024-02-10T07:29:48.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]: dispatch 2024-02-10T07:29:48.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/96497648' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2572554528"}]': finished 2024-02-10T07:29:48.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[29477]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T07:29:48.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2404291763' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]: dispatch 2024-02-10T07:29:48.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:47 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]: dispatch 2024-02-10T07:29:48.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:47 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/96497648' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2572554528"}]': finished 2024-02-10T07:29:48.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:47 smithi136 ceph-mon[25575]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T07:29:48.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:47 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2404291763' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]: dispatch 2024-02-10T07:29:48.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:47 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]: dispatch 2024-02-10T07:29:49.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 510 B/s wr, 132 op/s 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]': finished 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: Adding iSCSI gateway http://:@172.21.15.64:5000 to Dashboard 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi064"}]: dispatch 2024-02-10T07:29:49.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi064"}]: dispatch 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2110644445' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]: dispatch 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]: dispatch 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 510 B/s wr, 132 op/s 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]': finished 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: Adding iSCSI gateway http://:@172.21.15.64:5000 to Dashboard 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi064"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi064"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2110644445' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]: dispatch 2024-02-10T07:29:49.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:48 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]: dispatch 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 510 B/s wr, 132 op/s 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/963865334"}]': finished 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: Adding iSCSI gateway http://:@172.21.15.64:5000 to Dashboard 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T07:29:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi064"}]: dispatch 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi064"}]: dispatch 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2110644445' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]: dispatch 2024-02-10T07:29:49.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:48 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]: dispatch 2024-02-10T07:29:50.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:49 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]': finished 2024-02-10T07:29:50.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:49 smithi136 ceph-mon[25575]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T07:29:50.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:49 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2087459120' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]: dispatch 2024-02-10T07:29:50.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:49 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]: dispatch 2024-02-10T07:29:50.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:49 smithi136 ceph-mon[25575]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-02-10T07:29:50.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]': finished 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[24760]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2087459120' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]: dispatch 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]: dispatch 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[24760]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3390118538"}]': finished 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[29477]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2087459120' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]: dispatch 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]: dispatch 2024-02-10T07:29:50.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:49 smithi064 ceph-mon[29477]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-02-10T07:29:51.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:50 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]': finished 2024-02-10T07:29:51.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:50 smithi136 ceph-mon[25575]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T07:29:51.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:50 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1101772076' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]: dispatch 2024-02-10T07:29:51.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:50 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]: dispatch 2024-02-10T07:29:51.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]': finished 2024-02-10T07:29:51.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[24760]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T07:29:51.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1101772076' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]: dispatch 2024-02-10T07:29:51.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]: dispatch 2024-02-10T07:29:51.284 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2854614764"}]': finished 2024-02-10T07:29:51.285 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[29477]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T07:29:51.285 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1101772076' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]: dispatch 2024-02-10T07:29:51.285 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:50 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]: dispatch 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]': finished 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4291908099' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1756309217"}]: dispatch 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: Checking dashboard <-> RGW credentials 2024-02-10T07:29:52.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:52.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:51 smithi136 ceph-mon[25575]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-10T07:29:52.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]': finished 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4291908099' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1756309217"}]: dispatch 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: Checking dashboard <-> RGW credentials 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[24760]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1756309217"}]': finished 2024-02-10T07:29:52.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T07:29:52.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4291908099' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1756309217"}]: dispatch 2024-02-10T07:29:52.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:52.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: Checking dashboard <-> RGW credentials 2024-02-10T07:29:52.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:29:52.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:51 smithi064 ceph-mon[29477]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-10T07:29:53.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:52 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4291908099' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1756309217"}]': finished 2024-02-10T07:29:53.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:52 smithi136 ceph-mon[25575]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T07:29:53.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:52 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2047597808' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]: dispatch 2024-02-10T07:29:53.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:52 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]: dispatch 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4291908099' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1756309217"}]': finished 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[24760]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2047597808' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]: dispatch 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]: dispatch 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4291908099' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1756309217"}]': finished 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[29477]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T07:29:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2047597808' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]: dispatch 2024-02-10T07:29:53.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:52 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]: dispatch 2024-02-10T07:29:53.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:29:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:29:53.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:29:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:29:54.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]': finished 2024-02-10T07:29:54.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[24760]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T07:29:54.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/729235065' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2955412271"}]: dispatch 2024-02-10T07:29:54.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[24760]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-10T07:29:54.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]': finished 2024-02-10T07:29:54.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[29477]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T07:29:54.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/729235065' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2955412271"}]: dispatch 2024-02-10T07:29:54.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:53 smithi064 ceph-mon[29477]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-10T07:29:54.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:53 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2794690239"}]': finished 2024-02-10T07:29:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:53 smithi136 ceph-mon[25575]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T07:29:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:53 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/729235065' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2955412271"}]: dispatch 2024-02-10T07:29:54.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:53 smithi136 ceph-mon[25575]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-10T07:29:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/729235065' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2955412271"}]': finished 2024-02-10T07:29:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[24760]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T07:29:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1412284653' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2572554528"}]: dispatch 2024-02-10T07:29:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:29:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/729235065' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2955412271"}]': finished 2024-02-10T07:29:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[29477]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T07:29:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1412284653' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2572554528"}]: dispatch 2024-02-10T07:29:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:54 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:29:55.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/729235065' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/2955412271"}]': finished 2024-02-10T07:29:55.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:54 smithi136 ceph-mon[25575]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T07:29:55.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1412284653' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2572554528"}]: dispatch 2024-02-10T07:29:55.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:54 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:29:55.998 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:29:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:29:55] "GET /metrics HTTP/1.1" 200 214914 "" "Prometheus/2.33.4" 2024-02-10T07:29:56.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:55 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1412284653' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2572554528"}]': finished 2024-02-10T07:29:56.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:55 smithi064 ceph-mon[24760]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T07:29:56.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:55 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/292015018' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1586222082"}]: dispatch 2024-02-10T07:29:56.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:55 smithi064 ceph-mon[24760]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:56.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:56 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1412284653' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2572554528"}]': finished 2024-02-10T07:29:56.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:56 smithi064 ceph-mon[29477]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T07:29:56.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:56 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/292015018' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1586222082"}]: dispatch 2024-02-10T07:29:56.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:56 smithi064 ceph-mon[29477]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:56.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:56 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1412284653' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2572554528"}]': finished 2024-02-10T07:29:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:56 smithi136 ceph-mon[25575]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T07:29:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:56 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/292015018' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1586222082"}]: dispatch 2024-02-10T07:29:56.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:56 smithi136 ceph-mon[25575]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:29:57.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:57 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/292015018' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1586222082"}]': finished 2024-02-10T07:29:57.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:57 smithi064 ceph-mon[24760]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T07:29:57.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:57 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1349338719' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3353043261"}]: dispatch 2024-02-10T07:29:57.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:57 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/292015018' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1586222082"}]': finished 2024-02-10T07:29:57.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:57 smithi064 ceph-mon[29477]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T07:29:57.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:57 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1349338719' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3353043261"}]: dispatch 2024-02-10T07:29:57.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:57 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/292015018' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1586222082"}]': finished 2024-02-10T07:29:57.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:57 smithi136 ceph-mon[25575]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T07:29:57.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:57 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1349338719' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3353043261"}]: dispatch 2024-02-10T07:29:58.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1349338719' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3353043261"}]': finished 2024-02-10T07:29:58.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[24760]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T07:29:58.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1198919886' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1410740193"}]: dispatch 2024-02-10T07:29:58.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[24760]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:29:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1349338719' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3353043261"}]': finished 2024-02-10T07:29:58.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[29477]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T07:29:58.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1198919886' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1410740193"}]: dispatch 2024-02-10T07:29:58.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:58 smithi064 ceph-mon[29477]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:29:58.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:58 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1349338719' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3353043261"}]': finished 2024-02-10T07:29:58.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:58 smithi136 ceph-mon[25575]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T07:29:58.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:58 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1198919886' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1410740193"}]: dispatch 2024-02-10T07:29:58.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:58 smithi136 ceph-mon[25575]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:29:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:59 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1198919886' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1410740193"}]': finished 2024-02-10T07:29:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:59 smithi064 ceph-mon[24760]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T07:29:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:29:59 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1622670855' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/184722407"}]: dispatch 2024-02-10T07:29:59.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:59 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1198919886' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1410740193"}]': finished 2024-02-10T07:29:59.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:59 smithi064 ceph-mon[29477]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T07:29:59.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:29:59 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1622670855' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/184722407"}]: dispatch 2024-02-10T07:29:59.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:59 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1198919886' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1410740193"}]': finished 2024-02-10T07:29:59.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:59 smithi136 ceph-mon[25575]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T07:29:59.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:29:59 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1622670855' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/184722407"}]: dispatch 2024-02-10T07:30:00.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1622670855' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/184722407"}]': finished 2024-02-10T07:30:00.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[24760]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T07:30:00.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/235945713' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/3057373934"}]: dispatch 2024-02-10T07:30:00.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[24760]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:00.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[24760]: overall HEALTH_OK 2024-02-10T07:30:00.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1622670855' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/184722407"}]': finished 2024-02-10T07:30:00.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[29477]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T07:30:00.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/235945713' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/3057373934"}]: dispatch 2024-02-10T07:30:00.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[29477]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:00.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:00 smithi064 ceph-mon[29477]: overall HEALTH_OK 2024-02-10T07:30:00.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:00 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1622670855' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/184722407"}]': finished 2024-02-10T07:30:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:00 smithi136 ceph-mon[25575]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T07:30:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:00 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/235945713' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/3057373934"}]: dispatch 2024-02-10T07:30:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:00 smithi136 ceph-mon[25575]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:00.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:00 smithi136 ceph-mon[25575]: overall HEALTH_OK 2024-02-10T07:30:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:01 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/235945713' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/3057373934"}]': finished 2024-02-10T07:30:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:01 smithi064 ceph-mon[24760]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T07:30:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:01 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2503886116' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/3057373934"}]: dispatch 2024-02-10T07:30:01.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:01 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/235945713' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/3057373934"}]': finished 2024-02-10T07:30:01.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:01 smithi064 ceph-mon[29477]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T07:30:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:01 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2503886116' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/3057373934"}]: dispatch 2024-02-10T07:30:01.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:01 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/235945713' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/3057373934"}]': finished 2024-02-10T07:30:01.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:01 smithi136 ceph-mon[25575]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T07:30:01.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:01 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2503886116' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/3057373934"}]: dispatch 2024-02-10T07:30:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:02 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2503886116' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/3057373934"}]': finished 2024-02-10T07:30:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:02 smithi064 ceph-mon[24760]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T07:30:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:02 smithi064 ceph-mon[24760]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:02.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:02 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2503886116' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/3057373934"}]': finished 2024-02-10T07:30:02.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:02 smithi064 ceph-mon[29477]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T07:30:02.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:02 smithi064 ceph-mon[29477]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:02.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:30:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:30:02.477 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:02 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2503886116' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/3057373934"}]': finished 2024-02-10T07:30:02.478 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:02 smithi136 ceph-mon[25575]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T07:30:02.478 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:02 smithi136 ceph-mon[25575]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:03.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:30:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:30:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:30:03.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:30:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:30:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:30:03.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:30:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:30:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:30:05.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:04 smithi064 ceph-mon[24760]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-02-10T07:30:05.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:04 smithi064 ceph-mon[29477]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-02-10T07:30:05.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:04 smithi136 ceph-mon[25575]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-02-10T07:30:06.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:05 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:06.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:05 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:06.065 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:30:05 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:05] "GET /metrics HTTP/1.1" 200 214914 "" "Prometheus/2.33.4" 2024-02-10T07:30:06.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:05 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:06 smithi064 ceph-mon[24760]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 0 op/s 2024-02-10T07:30:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:06 smithi064 ceph-mon[29477]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 0 op/s 2024-02-10T07:30:07.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:06 smithi136 ceph-mon[25575]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 0 op/s 2024-02-10T07:30:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:08 smithi064 ceph-mon[24760]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:09.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:08 smithi064 ceph-mon[29477]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:09.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:08 smithi136 ceph-mon[25575]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:10 smithi136 ceph-mon[25575]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:30:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:10 smithi064 ceph-mon[24760]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:30:11.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:10 smithi064 ceph-mon[29477]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:30:12.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:30:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:30:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:12 smithi064 ceph-mon[24760]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:30:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:12 smithi064 ceph-mon[29477]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:30:13.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:12 smithi136 ceph-mon[25575]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:30:13.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:30:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:30:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:30:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:14 smithi064 ceph-mon[24760]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:15.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:14 smithi064 ceph-mon[29477]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:15.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:14 smithi136 ceph-mon[25575]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:16.011 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:15 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:16.011 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:15 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:16.012 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:30:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:15] "GET /metrics HTTP/1.1" 200 214955 "" "Prometheus/2.33.4" 2024-02-10T07:30:16.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:15 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:16 smithi064 ceph-mon[24760]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:17.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:16 smithi064 ceph-mon[29477]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:17.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:16 smithi136 ceph-mon[25575]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:18 smithi064 ceph-mon[24760]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:19.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:18 smithi064 ceph-mon[29477]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:19.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:18 smithi136 ceph-mon[25575]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:20 smithi136 ceph-mon[25575]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:20 smithi064 ceph-mon[24760]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:21.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:20 smithi064 ceph-mon[29477]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:22.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:30:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:30:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:22 smithi064 ceph-mon[24760]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:23.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:22 smithi064 ceph-mon[29477]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:23.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:22 smithi136 ceph-mon[25575]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:24 smithi064 ceph-mon[24760]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:25.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:24 smithi064 ceph-mon[29477]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:25.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:24 smithi136 ceph-mon[25575]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:25.773 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:30:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:25] "GET /metrics HTTP/1.1" 200 214886 "" "Prometheus/2.33.4" 2024-02-10T07:30:26.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:25 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:26.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:25 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:25 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:26 smithi064 ceph-mon[24760]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:26 smithi064 ceph-mon[29477]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:26 smithi136 ceph-mon[25575]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T07:30:29.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:28 smithi064 ceph-mon[24760]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:29.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:28 smithi064 ceph-mon[29477]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:29.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:28 smithi136 ceph-mon[25575]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:30.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:30 smithi136 ceph-mon[25575]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:30 smithi064 ceph-mon[24760]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:31.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:30 smithi064 ceph-mon[29477]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:32.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:30:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:30:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:32 smithi064 ceph-mon[24760]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:33.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:32 smithi064 ceph-mon[29477]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:33.115 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:32 smithi136 ceph-mon[25575]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:35.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:34 smithi064 ceph-mon[24760]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:35.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:34 smithi064 ceph-mon[29477]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:35.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:34 smithi136 ceph-mon[25575]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:36.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:35] "GET /metrics HTTP/1.1" 200 214886 "" "Prometheus/2.33.4" 2024-02-10T07:30:36.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:30:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:30:36.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:35 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:30:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:30:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:30:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:30:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:36 smithi064 ceph-mon[24760]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:37.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:36 smithi064 ceph-mon[29477]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:37.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:36 smithi136 ceph-mon[25575]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:39.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:38 smithi064 ceph-mon[24760]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:39.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:38 smithi064 ceph-mon[29477]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:39.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:38 smithi136 ceph-mon[25575]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:40.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:40 smithi136 ceph-mon[25575]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:40 smithi064 ceph-mon[24760]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:41.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:40 smithi064 ceph-mon[29477]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:42.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:30:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:30:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:42 smithi064 ceph-mon[24760]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:42 smithi064 ceph-mon[29477]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:43.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:42 smithi136 ceph-mon[25575]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:44 smithi064 ceph-mon[24760]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:45.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:44 smithi064 ceph-mon[29477]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:45.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:44 smithi136 ceph-mon[25575]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:46.045 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:45 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:46.045 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:45 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:46.046 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:30:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:45] "GET /metrics HTTP/1.1" 200 214842 "" "Prometheus/2.33.4" 2024-02-10T07:30:46.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:45 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:46 smithi064 ceph-mon[24760]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:47.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:46 smithi064 ceph-mon[29477]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:47.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:46 smithi136 ceph-mon[25575]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:49.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:48 smithi064 ceph-mon[24760]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:49.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:48 smithi064 ceph-mon[29477]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:49.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:48 smithi136 ceph-mon[25575]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:50 smithi136 ceph-mon[25575]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:51.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:50 smithi064 ceph-mon[24760]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:50 smithi064 ceph-mon[29477]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:51.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:51 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:51.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:51 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:51.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:51 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:51.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:51 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:51.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:51 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:51.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:51 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:52.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:30:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:30:53.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:52 smithi136 ceph-mon[25575]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:52 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:30:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:52 smithi064 ceph-mon[24760]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:52 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:30:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:52 smithi064 ceph-mon[29477]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:52 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:30:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:54 smithi064 ceph-mon[24760]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:55.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:54 smithi064 ceph-mon[29477]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:55.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:54 smithi136 ceph-mon[25575]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:55.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:30:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:30:55] "GET /metrics HTTP/1.1" 200 214845 "" "Prometheus/2.33.4" 2024-02-10T07:30:56.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:55 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:56.116 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:55 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:56.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:55 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:30:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:56 smithi136 ceph-mon[25575]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:57.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:56 smithi064 ceph-mon[24760]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:57.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:56 smithi064 ceph-mon[29477]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:30:59.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:30:58 smithi136 ceph-mon[25575]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:30:58 smithi064 ceph-mon[24760]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:30:59.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:30:58 smithi064 ceph-mon[29477]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:00 smithi136 ceph-mon[25575]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:00 smithi064 ceph-mon[24760]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:01.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:00 smithi064 ceph-mon[29477]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:02.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:31:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:31:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:02 smithi064 ceph-mon[24760]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:03.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:02 smithi064 ceph-mon[29477]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:03.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:02 smithi136 ceph-mon[25575]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:05.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:04 smithi064 ceph-mon[24760]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:05.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:04 smithi064 ceph-mon[29477]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:05.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:04 smithi136 ceph-mon[25575]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:06.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:05 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:06.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:05 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:06.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:31:05 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:05] "GET /metrics HTTP/1.1" 200 214845 "" "Prometheus/2.33.4" 2024-02-10T07:31:06.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:05 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:06 smithi064 ceph-mon[24760]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:06 smithi064 ceph-mon[29477]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:07.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:06 smithi136 ceph-mon[25575]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:08 smithi064 ceph-mon[24760]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:09.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:08 smithi064 ceph-mon[29477]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:09.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:08 smithi136 ceph-mon[25575]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:10 smithi136 ceph-mon[25575]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:10 smithi064 ceph-mon[24760]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:11.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:10 smithi064 ceph-mon[29477]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:12.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:31:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:31:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:12 smithi064 ceph-mon[24760]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:12 smithi064 ceph-mon[29477]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:13.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:12 smithi136 ceph-mon[25575]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:14 smithi064 ceph-mon[24760]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:15.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:14 smithi064 ceph-mon[29477]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:15.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:14 smithi136 ceph-mon[25575]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:15 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:16.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:15 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:16.065 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:31:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:15] "GET /metrics HTTP/1.1" 200 214846 "" "Prometheus/2.33.4" 2024-02-10T07:31:16.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:15 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:17.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:16 smithi064 ceph-mon[24760]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:17.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:16 smithi064 ceph-mon[29477]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:17.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:16 smithi136 ceph-mon[25575]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:19.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:18 smithi064 ceph-mon[24760]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:19.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:18 smithi064 ceph-mon[29477]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:19.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:18 smithi136 ceph-mon[25575]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:20 smithi136 ceph-mon[25575]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:20 smithi064 ceph-mon[24760]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:21.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:20 smithi064 ceph-mon[29477]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:22.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:31:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:31:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:22 smithi064 ceph-mon[24760]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:23.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:22 smithi064 ceph-mon[29477]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:23.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:22 smithi136 ceph-mon[25575]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:24 smithi064 ceph-mon[24760]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:25.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:24 smithi064 ceph-mon[29477]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:25.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:24 smithi136 ceph-mon[25575]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:26.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:25 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:26.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:25 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:26.065 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:31:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:25] "GET /metrics HTTP/1.1" 200 214845 "" "Prometheus/2.33.4" 2024-02-10T07:31:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:25 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:26 smithi064 ceph-mon[24760]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:26 smithi064 ceph-mon[29477]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:26 smithi136 ceph-mon[25575]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:29.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:28 smithi064 ceph-mon[24760]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:29.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:28 smithi064 ceph-mon[29477]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:29.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:28 smithi136 ceph-mon[25575]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:30.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:30 smithi136 ceph-mon[25575]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:30 smithi064 ceph-mon[24760]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:30 smithi064 ceph-mon[29477]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:32.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:31:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:31:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:32 smithi064 ceph-mon[24760]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:33.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:32 smithi064 ceph-mon[29477]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:33.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:32 smithi136 ceph-mon[25575]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:35.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:34 smithi064 ceph-mon[24760]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:35.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:34 smithi064 ceph-mon[29477]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:35.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:34 smithi136 ceph-mon[25575]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:36.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:35] "GET /metrics HTTP/1.1" 200 214845 "" "Prometheus/2.33.4" 2024-02-10T07:31:36.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:36.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:31:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:31:36.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:35 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:31:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:31:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:31:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:31:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:36 smithi064 ceph-mon[24760]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:37.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:36 smithi064 ceph-mon[29477]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:37.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:36 smithi136 ceph-mon[25575]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:39.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:38 smithi064 ceph-mon[24760]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:39.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:38 smithi064 ceph-mon[29477]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:39.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:38 smithi136 ceph-mon[25575]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:40.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T07:31:40.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:40 smithi064 ceph-mon[24760]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:40.746 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:40 smithi064 ceph-mon[29477]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:40.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:40 smithi136 ceph-mon[25575]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:42.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:31:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:31:42.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T07:31:42.998 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:42 smithi064 ceph-mon[24760]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:42.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:42 smithi064 ceph-mon[29477]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:43.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:42 smithi136 ceph-mon[25575]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:45.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:44 smithi064 ceph-mon[24760]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:45.010 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:44 smithi064 ceph-mon[29477]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:45.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:44 smithi136 ceph-mon[25575]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:45.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T07:31:45.799 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:31:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:45] "GET /metrics HTTP/1.1" 200 214867 "" "Prometheus/2.33.4" 2024-02-10T07:31:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:45 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:46.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:45 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:46.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:45 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:46.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:46 smithi064 ceph-mon[24760]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:46.938 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:46 smithi064 ceph-mon[29477]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:47.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:46 smithi136 ceph-mon[25575]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:47.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T07:31:48.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:48 smithi064 ceph-mon[24760]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:48.989 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:48 smithi064 ceph-mon[29477]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:49.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:48 smithi136 ceph-mon[25575]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:49.413 INFO:teuthology.orchestra.run.smithi064.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:49.985 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T07:31:49.994 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T07:31:49.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-10T07:31:50.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[24760]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:50.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:31:50.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:50.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:50.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:31:50.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:50.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:50 smithi064 ceph-mon[29477]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:50.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:31:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:50 smithi136 ceph-mon[25575]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:52.243 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:31:52.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:31:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (2m) 2m ago 3m 15.5M - ba2b418f427c 9e05d1c8d880 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (2m) 2m ago 2m 47.5M - 8.3.5 dad864ee21e9 1d718a66eb18 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (2m) 2m ago 2m 74.7M - 3.5 e1d6a67b021e 517742b65a2b 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443 running (7m) 2m ago 7m 411M - 17.2.0 e1d6a67b021e ebf96559d228 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 49c504cf1f60 2024-02-10T07:31:53.046 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (9m) 2m ago 9m 47.6M 2048M 17.2.0 e1d6a67b021e 0975e381bf03 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (7m) 2m ago 7m 39.7M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (8m) 2m ago 8m 44.1M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (3m) 2m ago 3m 17.5M - 1dbe0e931976 377b0b02dd9f 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (3m) 2m ago 3m 16.5M - 1dbe0e931976 bd193e88eb53 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (7m) 2m ago 7m 52.0M 4016M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (7m) 2m ago 7m 52.2M 4016M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (6m) 2m ago 6m 48.5M 4016M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (6m) 2m ago 6m 50.9M 4016M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (5m) 2m ago 5m 47.6M 4272M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (5m) 2m ago 5m 45.3M 4272M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (5m) 2m ago 5m 46.4M 4272M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (4m) 2m ago 4m 49.8M 4272M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (2m) 2m ago 3m 44.4M - 514e6a882f6e 75ee28ab375d 2024-02-10T07:31:53.047 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (2m) 2m ago 2m 70.1M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:31:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:31:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (2m) 2m ago 2m 68.7M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:31:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (2m) 2m ago 2m 68.6M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:31:53.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:52 smithi064 ceph-mon[24760]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:53.048 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:52 smithi064 ceph-mon[29477]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:53.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:52 smithi136 ceph-mon[25575]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:53.502 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:31:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:31:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-10T07:31:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:31:53.504 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:31:53.800 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:53 smithi064 ceph-mon[24760]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:53.800 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:53 smithi064 ceph-mon[24760]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:53.800 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:53 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/808441332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:31:53.800 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:53 smithi064 ceph-mon[29477]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:53.800 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:53 smithi064 ceph-mon[29477]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:53.801 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:53 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/808441332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [], 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "", 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 image" 2024-02-10T07:31:53.893 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:31:53.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:53 smithi136 ceph-mon[25575]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:53.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:53 smithi136 ceph-mon[25575]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:53.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:53 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/808441332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:31:54.418 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:31:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[24760]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[24760]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[24760]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2798840269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:31:55.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[29477]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[29477]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[29477]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:54 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2798840269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:31:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:54 smithi136 ceph-mon[25575]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:54 smithi136 ceph-mon[25575]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:54 smithi136 ceph-mon[25575]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:55.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2798840269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:31:56.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:55 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:56.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:31:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:31:55] "GET /metrics HTTP/1.1" 200 214840 "" "Prometheus/2.33.4" 2024-02-10T07:31:56.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:55 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:56.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:55 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:31:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:56 smithi064 ceph-mon[24760]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:57.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:56 smithi064 ceph-mon[29477]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:56 smithi136 ceph-mon[25575]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:31:59.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:31:58 smithi064 ceph-mon[24760]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:59.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:31:58 smithi064 ceph-mon[29477]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:31:59.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:31:58 smithi136 ceph-mon[25575]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:00 smithi136 ceph-mon[25575]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:00 smithi064 ceph-mon[24760]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:01.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:00 smithi064 ceph-mon[29477]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:02.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:32:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:02 smithi064 ceph-mon[24760]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:03.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:02 smithi064 ceph-mon[29477]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:03.118 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:02 smithi136 ceph-mon[25575]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:05.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:04 smithi064 ceph-mon[24760]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:05.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:04 smithi064 ceph-mon[29477]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:05.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:04 smithi136 ceph-mon[25575]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:06.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:05 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:06.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:05 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:06.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:32:05 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:05] "GET /metrics HTTP/1.1" 200 214840 "" "Prometheus/2.33.4" 2024-02-10T07:32:06.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:05 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:07.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:06 smithi064 ceph-mon[24760]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:07.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:06 smithi064 ceph-mon[29477]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:07.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:06 smithi136 ceph-mon[25575]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:08 smithi064 ceph-mon[24760]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:09.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:08 smithi064 ceph-mon[29477]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:09.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:08 smithi136 ceph-mon[25575]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:10 smithi136 ceph-mon[25575]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:10 smithi064 ceph-mon[24760]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:11.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:10 smithi064 ceph-mon[29477]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:12.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:32:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:12 smithi064 ceph-mon[24760]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:12 smithi064 ceph-mon[29477]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:13.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:12 smithi136 ceph-mon[25575]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:15.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:14 smithi064 ceph-mon[24760]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:15.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:14 smithi064 ceph-mon[29477]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:15.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:14 smithi136 ceph-mon[25575]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:15.808 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:32:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:15] "GET /metrics HTTP/1.1" 200 214820 "" "Prometheus/2.33.4" 2024-02-10T07:32:16.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:15 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:16.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:15 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:16.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:15 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:17.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:16 smithi136 ceph-mon[25575]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[24760]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:17.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:16 smithi064 ceph-mon[29477]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-10T07:32:18.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:17 smithi136 ceph-mon[25575]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on smithi136 2024-02-10T07:32:18.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:17 smithi064 ceph-mon[24760]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on smithi136 2024-02-10T07:32:18.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:17 smithi064 ceph-mon[29477]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on smithi136 2024-02-10T07:32:19.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:18 smithi136 ceph-mon[25575]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:18 smithi064 ceph-mon[24760]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:19.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:18 smithi064 ceph-mon[29477]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:20 smithi136 ceph-mon[25575]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:20 smithi064 ceph-mon[24760]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:21.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:20 smithi064 ceph-mon[29477]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:22.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:32:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:22 smithi064 ceph-mon[24760]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:22 smithi064 ceph-mon[29477]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:23.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:22 smithi136 ceph-mon[25575]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:32:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:32:24.755 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:32:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:24 smithi064 ceph-mon[24760]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:25.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:24 smithi064 ceph-mon[29477]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:25.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:24 smithi136 ceph-mon[25575]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:25.361 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:32:25.361 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (3m) 2m ago 3m 15.5M - ba2b418f427c 9e05d1c8d880 2024-02-10T07:32:25.361 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (3m) 2m ago 3m 47.5M - 8.3.5 dad864ee21e9 1d718a66eb18 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (2m) 2m ago 2m 74.7M - 3.5 e1d6a67b021e 517742b65a2b 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e ebf96559d228 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 49c504cf1f60 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (10m) 2m ago 10m 47.6M 2048M 17.2.0 e1d6a67b021e 0975e381bf03 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (8m) 2m ago 8m 39.7M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (8m) 2m ago 8m 44.1M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (4m) 2m ago 4m 17.5M - 1dbe0e931976 377b0b02dd9f 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (4m) 2m ago 4m 16.5M - 1dbe0e931976 bd193e88eb53 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (7m) 2m ago 7m 52.0M 4016M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (7m) 2m ago 7m 52.2M 4016M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (7m) 2m ago 7m 48.5M 4016M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (6m) 2m ago 6m 50.9M 4016M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (6m) 2m ago 6m 47.6M 4272M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:32:25.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (5m) 2m ago 5m 45.3M 4272M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (5m) 2m ago 5m 46.4M 4272M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (5m) 2m ago 5m 49.8M 4272M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (3m) 2m ago 4m 44.4M - 514e6a882f6e 75ee28ab375d 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (2m) 2m ago 2m 70.1M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (2m) 2m ago 2m 68.7M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:32:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (2m) 2m ago 2m 68.6M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:32:25.722 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:32:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-10T07:32:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:32:25.724 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:32:26.030 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:32:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:32:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:32:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [], 2024-02-10T07:32:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "0/23 daemons upgraded", 2024-02-10T07:32:26.031 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 image on host smithi136" 2024-02-10T07:32:26.031 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:32:26.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-mon[24760]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:26.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:26.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1028440818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:32:26.032 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:25] "GET /metrics HTTP/1.1" 200 214777 "" "Prometheus/2.33.4" 2024-02-10T07:32:26.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-mon[29477]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:26.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:26.032 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:25 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1028440818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:32:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:25 smithi136 ceph-mon[25575]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:25 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:26.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:25 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1028440818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:32:26.397 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:32:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[29477]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[29477]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[29477]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:27.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4206919065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:32:27.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[24760]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:27.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[24760]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:27.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[24760]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:27.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:26 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4206919065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:32:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:26 smithi136 ceph-mon[25575]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:26 smithi136 ceph-mon[25575]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:26 smithi136 ceph-mon[25575]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:26 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4206919065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:32:28.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:27 smithi064 ceph-mon[24760]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:28.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:27 smithi064 ceph-mon[29477]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:28.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:27 smithi136 ceph-mon[25575]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:29.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:28 smithi064 ceph-mon[24760]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:29.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:28 smithi064 ceph-mon[29477]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:29.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:28 smithi136 ceph-mon[25575]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:30.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:30 smithi136 ceph-mon[25575]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:31.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:30 smithi064 ceph-mon[24760]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:31.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:30 smithi064 ceph-mon[29477]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:32.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:32:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:32 smithi064 ceph-mon[24760]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:33.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:32 smithi064 ceph-mon[29477]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:33.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:32 smithi136 ceph-mon[25575]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:33.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:32:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:32:33.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:32:33.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:32:35.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:34 smithi064 ceph-mon[24760]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:35.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:34 smithi064 ceph-mon[29477]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:35.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:34 smithi136 ceph-mon[25575]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:36.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:35] "GET /metrics HTTP/1.1" 200 214777 "" "Prometheus/2.33.4" 2024-02-10T07:32:36.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:36.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:36.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:35 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:36.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:35 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:36.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:35 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:36.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:35 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:36 smithi064 ceph-mon[24760]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:36 smithi064 ceph-mon[29477]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:37.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:36 smithi136 ceph-mon[25575]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:39.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:38 smithi064 ceph-mon[24760]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:39.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:38 smithi064 ceph-mon[29477]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:39.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:38 smithi136 ceph-mon[25575]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:40.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:40 smithi136 ceph-mon[25575]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:40 smithi064 ceph-mon[24760]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:41.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:40 smithi064 ceph-mon[29477]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:42.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[26755]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:32:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:42 smithi064 ceph-mon[24760]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:43.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:42 smithi064 ceph-mon[29477]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:43.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:42 smithi136 ceph-mon[25575]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:43.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:32:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:32:43.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:32:43.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:32:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:44 smithi064 ceph-mon[24760]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:45.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:44 smithi064 ceph-mon[29477]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:45.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:44 smithi136 ceph-mon[25575]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:45 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:46.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:32:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:45] "GET /metrics HTTP/1.1" 200 214786 "" "Prometheus/2.33.4" 2024-02-10T07:32:46.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:45 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:46.087 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:45 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:47.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:46 smithi136 ceph-mon[25575]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:47.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:46 smithi064 ceph-mon[24760]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:47.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:46 smithi064 ceph-mon[29477]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:48.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: Upgrade: Updating mgr.x 2024-02-10T07:32:48.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:48.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:32:48.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:32:48.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:32:48.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:48.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: Deploying daemon mgr.x on smithi136 2024-02-10T07:32:48.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:48 smithi136 ceph-mon[25575]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:48.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: Upgrade: Updating mgr.x 2024-02-10T07:32:48.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:48.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: Deploying daemon mgr.x on smithi136 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[24760]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: Upgrade: Updating mgr.x 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:32:48.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:48.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: Deploying daemon mgr.x on smithi136 2024-02-10T07:32:48.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:48 smithi064 ceph-mon[29477]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:48.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:48 smithi136 systemd[1]: Stopping Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:32:48.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:48 smithi136 bash[46568]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T07:32:49.375 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 podman[46581]: 2024-02-10 07:32:49.094714926 +0000 UTC m=+0.158254390 container died ebf96559d2289b4a9cb5f3ce16ff2396a03cd8a3dab59037d899ae973dc3800b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, ceph=True, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, architecture=x86_64, release=754, io.openshift.expose-services=, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870) 2024-02-10T07:32:49.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 podman[46581]: 2024-02-10 07:32:49.540492041 +0000 UTC m=+0.604031545 container cleanup ebf96559d2289b4a9cb5f3ce16ff2396a03cd8a3dab59037d899ae973dc3800b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, ceph=True, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8) 2024-02-10T07:32:49.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 bash[46581]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x 2024-02-10T07:32:50.064 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:49 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:49.771Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 podman[46595]: 2024-02-10 07:32:49.738487628 +0000 UTC m=+0.639278143 container remove ebf96559d2289b4a9cb5f3ce16ff2396a03cd8a3dab59037d899ae973dc3800b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, architecture=x86_64, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 bash[46608]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 systemd[1]: Stopped Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Consumed 16.473s CPU time. 2024-02-10T07:32:50.191 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:49 smithi136 systemd[1]: Starting Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:32:50.470 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 podman[46712]: 2024-02-10 07:32:50.09369867 +0000 UTC m=+0.018084277 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:32:50.470 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 podman[46712]: 2024-02-10 07:32:50.209290328 +0000 UTC m=+0.133675922 container create 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T07:32:50.470 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 podman[46712]: 2024-02-10 07:32:50.470252555 +0000 UTC m=+0.394638147 container init 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T07:32:50.721 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 podman[46712]: 2024-02-10 07:32:50.473436015 +0000 UTC m=+0.397821614 container start 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T07:32:50.721 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 ceph-mgr[46731]: -- 172.21.15.136:0/2906184056 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563be70e01a0 con 0x563be70b6800 2024-02-10T07:32:50.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 bash[46712]: 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 2024-02-10T07:32:50.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:50 smithi136 systemd[1]: Started Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:32:50.955 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[29477]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:50.955 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:50.955 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:50.955 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:32:50.955 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[24760]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:50.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:50.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:50.956 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:50 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:32:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:50 smithi136 ceph-mon[25575]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:50 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:50 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:50 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:32:51.325 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:51.063+0000 7f6dcb6da200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:32:51.325 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:51.156+0000 7f6dcb6da200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:32:51.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:51.847+0000 7f6dcb6da200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:32:51.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:51.947+0000 7f6dcb6da200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:32:52.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.109+0000 7f6dcb6da200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:32:52.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.212+0000 7f6dcb6da200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:32:52.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.298+0000 7f6dcb6da200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:32:52.725 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.594+0000 7f6dcb6da200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:32:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:52 smithi064 ceph-mon[24760]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:53.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:52 smithi064 ceph-mon[29477]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:53.073 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-mon[25575]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:53.073 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.724+0000 7f6dcb6da200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:32:53.073 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.816+0000 7f6dcb6da200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:32:53.073 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:52.915+0000 7f6dcb6da200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:32:53.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:53.104+0000 7f6dcb6da200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:32:53.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:32:53.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:32:53.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:32:53.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:32:54.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:54.108+0000 7f6dcb6da200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:32:54.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:54.254+0000 7f6dcb6da200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:32:54.889 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:54 smithi136 ceph-mon[25575]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:54.889 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:54.592+0000 7f6dcb6da200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:32:54.889 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:54.683+0000 7f6dcb6da200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:32:54.890 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:54.888+0000 7f6dcb6da200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:32:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:54 smithi064 ceph-mon[24760]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:55.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:54 smithi064 ceph-mon[29477]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:55.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:55.082+0000 7f6dcb6da200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:32:55.665 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:55.389+0000 7f6dcb6da200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:32:55.922 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:55.668+0000 7f6dcb6da200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:32:55.922 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:55 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:55.922 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:55 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:56.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:32:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:32:55] "GET /metrics HTTP/1.1" 200 214778 "" "Prometheus/2.33.4" 2024-02-10T07:32:56.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:55 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:56.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:55 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:56.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:55 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:56.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:55 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:56.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:55.921+0000 7f6dcb6da200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:32:56.727 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:32:56.943 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:56 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:56.764+0000 7f6dcb6da200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:32:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:56 smithi064 ceph-mon[24760]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:56 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:57.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:56 smithi064 ceph-mon[29477]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:57.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:56 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:56 smithi136 ceph-mon[25575]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:32:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:56 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:32:57.326 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (3m) 3m ago 4m 15.5M - ba2b418f427c 9e05d1c8d880 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (4m) 2s ago 3m 58.2M - 8.3.5 dad864ee21e9 1d718a66eb18 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (3m) 3m ago 3m 74.7M - 3.5 e1d6a67b021e 517742b65a2b 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283 running (7s) 2s ago 8m 49.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 76cb04fd4e2b 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e 49c504cf1f60 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (10m) 3m ago 10m 47.6M 2048M 17.2.0 e1d6a67b021e 0975e381bf03 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (9m) 2s ago 9m 37.6M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (9m) 3m ago 9m 44.1M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (4m) 3m ago 4m 17.5M - 1dbe0e931976 377b0b02dd9f 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (4m) 2s ago 4m 16.4M - 1dbe0e931976 bd193e88eb53 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (8m) 3m ago 8m 52.0M 4016M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (8m) 3m ago 8m 52.2M 4016M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (7m) 3m ago 7m 48.5M 4016M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (7m) 3m ago 7m 50.9M 4016M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:32:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (6m) 2s ago 6m 50.5M 4272M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (6m) 2s ago 6m 48.2M 4272M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (6m) 2s ago 6m 48.9M 4272M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (5m) 2s ago 5m 51.7M 4272M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (3m) 2s ago 5m 66.7M - 514e6a882f6e 75ee28ab375d 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (3m) 3m ago 3m 70.1M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (3m) 2s ago 3m 70.5M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (3m) 3m ago 3m 68.7M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:32:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (3m) 2s ago 3m 69.5M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:32:57.686 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:57.351+0000 7f6dcb6da200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:32:57.686 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:57.463+0000 7f6dcb6da200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:32:57.686 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:57.685+0000 7f6dcb6da200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:32:57.687 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:32:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:32:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-10T07:32:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:32:57.689 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:32:57.953 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:32:57] ENGINE Bus STARTING 2024-02-10T07:32:57.954 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: CherryPy Checker: 2024-02-10T07:32:57.954 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: The Application mounted at '' has an empty config. 2024-02-10T07:32:57.954 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:32:57.954 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:32:57] ENGINE Serving on http://:::9283 2024-02-10T07:32:57.954 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:32:57] ENGINE Bus STARTED 2024-02-10T07:32:58.000 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:32:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:32:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:32:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [], 2024-02-10T07:32:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T07:32:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading mgr daemons" 2024-02-10T07:32:58.001 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:32:58.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:58.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2490466266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:32:58.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: Standby manager daemon x restarted 2024-02-10T07:32:58.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: Standby manager daemon x started 2024-02-10T07:32:58.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:32:58.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:32:58.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:32:58.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:57 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2490466266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: Standby manager daemon x restarted 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: Standby manager daemon x started 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:32:58.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2490466266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: Standby manager daemon x restarted 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: Standby manager daemon x started 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:32:58.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:57 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:32:58.362 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:32:58.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:32:58.385Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:32:59.137 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:58 smithi136 ceph-mon[25575]: from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:59.138 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:58 smithi136 ceph-mon[25575]: from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:59.138 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:58 smithi136 ceph-mon[25575]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:59.138 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:58 smithi136 ceph-mon[25575]: mgrmap e21: y(active, since 5m), standbys: x 2024-02-10T07:32:59.138 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:58 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2038925853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:32:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[24760]: from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[24760]: from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[24760]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:59.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[24760]: mgrmap e21: y(active, since 5m), standbys: x 2024-02-10T07:32:59.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2038925853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:32:59.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[29477]: from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:59.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[29477]: from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:59.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[29477]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:59.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[29477]: mgrmap e21: y(active, since 5m), standbys: x 2024-02-10T07:32:59.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:58 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2038925853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:33:00.212 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:33:00] ENGINE Bus STOPPING 2024-02-10T07:33:00.212 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:33:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:33:00.212 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:33:00] ENGINE Bus STOPPED 2024-02-10T07:33:00.212 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:00.181+0000 7f96c2b1f700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T07:33:00.212 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:00.212 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:32:59 smithi136 ceph-mon[25575]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-10T07:33:00.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[24760]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: from='mgr.14421 172.21.15.64:0/4207231134' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-10T07:33:00.214 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:32:59 smithi064 ceph-mon[29477]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T07:33:00.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:33:00] ENGINE Bus STARTING 2024-02-10T07:33:00.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:33:00] ENGINE Serving on http://:::9283 2024-02-10T07:33:00.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:33:00] ENGINE Bus STARTED 2024-02-10T07:33:00.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setuser ceph since I am not root 2024-02-10T07:33:00.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ignoring --setgroup ceph since I am not root 2024-02-10T07:33:00.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mgr[24998]: -- 172.21.15.64:0/63816898 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568e8676340 con 0x5568e864ec00 2024-02-10T07:33:01.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:00.710+0000 7fe87b388000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:33:01.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:00.808+0000 7fe87b388000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:33:01.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: Failing over to other MGR 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: mgrmap e22: x(active, starting, since 0.991676s) 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:33:01.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: Manager daemon x is now available 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: Queued rgw.foo for migration 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: Queued rgw.smpl for migration 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: Failing over to other MGR 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: mgrmap e22: x(active, starting, since 0.991676s) 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:33:01.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: Manager daemon x is now available 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: Queued rgw.foo for migration 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: Queued rgw.smpl for migration 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:33:01.069 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T07:33:01.070 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T07:33:01.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:33:01.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:33:01.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T07:33:01.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:00 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T07:33:01.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-10T07:33:01.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: Failing over to other MGR 2024-02-10T07:33:01.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:01.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: mgrmap e22: x(active, starting, since 0.991676s) 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:33:01.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: Manager daemon x is now available 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: Queued rgw.foo for migration 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: Queued rgw.smpl for migration 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:01.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:01.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:01.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:33:01.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:33:01.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T07:33:01.227 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:00 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-10T07:33:01.751 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:01.422+0000 7fe87b388000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:33:01.751 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:01.513+0000 7fe87b388000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:33:01.751 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:01.598+0000 7fe87b388000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:33:01.751 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:01.749+0000 7fe87b388000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:33:02.030 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:01.832+0000 7fe87b388000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:33:02.030 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:01.944+0000 7fe87b388000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:33:02.030 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:02.029+0000 7fe87b388000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:33:02.286 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:02.197+0000 7fe87b388000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:33:02.286 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[29477]: Deploying cephadm binary to smithi064 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[29477]: Deploying cephadm binary to smithi136 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[29477]: mgrmap e23: x(active, since 1.99416s) 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[29477]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[24760]: Deploying cephadm binary to smithi064 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[24760]: Deploying cephadm binary to smithi136 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[24760]: mgrmap e23: x(active, since 1.99416s) 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[24760]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:02.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:02.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:01] "GET /metrics HTTP/1.1" 200 35154 "" "Prometheus/2.33.4" 2024-02-10T07:33:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:02 smithi136 ceph-mon[25575]: Deploying cephadm binary to smithi064 2024-02-10T07:33:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:02 smithi136 ceph-mon[25575]: Deploying cephadm binary to smithi136 2024-02-10T07:33:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:02 smithi136 ceph-mon[25575]: mgrmap e23: x(active, since 1.99416s) 2024-02-10T07:33:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:02 smithi136 ceph-mon[25575]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:02 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:02.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:02 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:02.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:02 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:02.285+0000 7fe87b388000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:33:03.676 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:03.342+0000 7fe87b388000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:33:03.676 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:03.476+0000 7fe87b388000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:33:03.676 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:03.674+0000 7fe87b388000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:33:03.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[24760]: [10/Feb/2024:07:33:01] ENGINE Bus STARTING 2024-02-10T07:33:03.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[24760]: [10/Feb/2024:07:33:01] ENGINE Serving on https://172.21.15.136:7150 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[24760]: [10/Feb/2024:07:33:02] ENGINE Serving on http://172.21.15.136:8765 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[24760]: [10/Feb/2024:07:33:02] ENGINE Bus STARTED 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[24760]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[24760]: mgrmap e24: x(active, since 3s) 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[29477]: [10/Feb/2024:07:33:01] ENGINE Bus STARTING 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[29477]: [10/Feb/2024:07:33:01] ENGINE Serving on https://172.21.15.136:7150 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[29477]: [10/Feb/2024:07:33:02] ENGINE Serving on http://172.21.15.136:8765 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[29477]: [10/Feb/2024:07:33:02] ENGINE Bus STARTED 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[29477]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:03.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-mon[29477]: mgrmap e24: x(active, since 3s) 2024-02-10T07:33:03.678 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:33:03.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:33:03.678 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:33:03.678 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:33:03.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:03 smithi136 ceph-mon[25575]: [10/Feb/2024:07:33:01] ENGINE Bus STARTING 2024-02-10T07:33:03.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:03 smithi136 ceph-mon[25575]: [10/Feb/2024:07:33:01] ENGINE Serving on https://172.21.15.136:7150 2024-02-10T07:33:03.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:03 smithi136 ceph-mon[25575]: [10/Feb/2024:07:33:02] ENGINE Serving on http://172.21.15.136:8765 2024-02-10T07:33:03.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:03 smithi136 ceph-mon[25575]: [10/Feb/2024:07:33:02] ENGINE Bus STARTED 2024-02-10T07:33:03.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:03 smithi136 ceph-mon[25575]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:03.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:03 smithi136 ceph-mon[25575]: mgrmap e24: x(active, since 3s) 2024-02-10T07:33:04.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:03.766+0000 7fe87b388000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:33:04.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:03.909+0000 7fe87b388000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:33:04.559 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:04.236+0000 7fe87b388000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:33:04.559 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:04.469+0000 7fe87b388000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:33:04.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:04.558+0000 7fe87b388000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:33:04.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:04.736+0000 7fe87b388000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:33:05.696 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:05 smithi064 ceph-mon[29477]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:05.696 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:05 smithi064 ceph-mon[29477]: mgrmap e25: x(active, since 5s) 2024-02-10T07:33:05.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:05 smithi064 ceph-mon[24760]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:05.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:05 smithi064 ceph-mon[24760]: mgrmap e25: x(active, since 5s) 2024-02-10T07:33:05.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:05 smithi136 ceph-mon[25575]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:05.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:05 smithi136 ceph-mon[25575]: mgrmap e25: x(active, since 5s) 2024-02-10T07:33:06.030 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:05 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:05.695+0000 7fe87b388000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:33:06.296 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:06.029+0000 7fe87b388000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:33:06.296 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:06.129+0000 7fe87b388000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:33:06.296 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:06.295+0000 7fe87b388000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:33:06.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:06 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:06 smithi136 ceph-mon[25575]: Standby manager daemon y started 2024-02-10T07:33:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:06 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T07:33:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:06 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:33:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:06 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T07:33:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:06 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:33:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[24760]: Standby manager daemon y started 2024-02-10T07:33:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T07:33:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:33:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T07:33:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[29477]: Standby manager daemon y started 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/2092357047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:33:06] ENGINE Bus STARTING 2024-02-10T07:33:06.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: CherryPy Checker: 2024-02-10T07:33:06.566 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: The Application mounted at '' has an empty config. 2024-02-10T07:33:06.566 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: 2024-02-10T07:33:06.566 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:33:06] ENGINE Serving on http://:::9283 2024-02-10T07:33:06.566 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: [10/Feb/2024:07:33:06] ENGINE Bus STARTED 2024-02-10T07:33:07.708 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:07 smithi136 ceph-mon[25575]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:07.708 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:07 smithi136 ceph-mon[25575]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-10T07:33:07.708 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:07 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:33:07.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:07 smithi064 ceph-mon[24760]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:07.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:07 smithi064 ceph-mon[24760]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-10T07:33:07.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:07 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:33:07.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:07 smithi064 ceph-mon[29477]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:33:07.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:07 smithi064 ceph-mon[29477]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-10T07:33:07.763 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:07 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:33:08.440 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:08.111Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:33:09.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:09 smithi136 ceph-mon[25575]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T07:33:09.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:09 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:09.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:09 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:09 smithi064 ceph-mon[24760]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T07:33:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:09 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:09 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:09 smithi064 ceph-mon[29477]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T07:33:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:09 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:09 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi064 to 2736M 2024-02-10T07:33:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:10.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:10 smithi136 ceph-mon[25575]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:33:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi064 to 2736M 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[24760]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:11.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi064 to 2736M 2024-02-10T07:33:11.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:10 smithi064 ceph-mon[29477]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:33:11.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:11 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:11.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:11 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:12.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:11 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:12.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:11 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:12.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:11 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:12.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:11 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:12.274 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:11] "GET /metrics HTTP/1.1" 200 35154 "" "Prometheus/2.33.4" 2024-02-10T07:33:12.692 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.64:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.136:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:33:12.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:12 smithi136 ceph-mon[25575]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T07:33:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:12 smithi064 ceph-mon[24760]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T07:33:13.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:12 smithi064 ceph-mon[29477]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T07:33:13.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:33:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:33:13.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:33:13.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:33:15.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:15 smithi064 ceph-mon[24760]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:15.319 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:15 smithi064 ceph-mon[29477]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:15 smithi136 ceph-mon[25575]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:16.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:33:16.413 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:16 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:16.413 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:16 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:16 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:16 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:16.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:16 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:16.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:16 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:17.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:17 smithi064 ceph-mon[24760]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:17.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:17 smithi064 ceph-mon[29477]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:17.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:17 smithi136 ceph-mon[25575]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:19.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:19 smithi136 ceph-mon[25575]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:19 smithi064 ceph-mon[24760]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:19 smithi064 ceph-mon[29477]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:33:21.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:20 smithi064 ceph-mon[24760]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:21.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:20 smithi064 ceph-mon[29477]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:21.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:20 smithi136 ceph-mon[25575]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:22.314 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:21] "GET /metrics HTTP/1.1" 200 38147 "" "Prometheus/2.33.4" 2024-02-10T07:33:22.590 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.64:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.136:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.511 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: Adjusting osd_memory_target on smithi136 to 2992M 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:33:23.512 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: Adjusting osd_memory_target on smithi136 to 2992M 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:23.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:33:23.514 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:33:23.514 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:33:23.514 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=error ts=2024-02-10T07:33:23.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:33:23.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: Adjusting osd_memory_target on smithi136 to 2992M 2024-02-10T07:33:23.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:23.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:23.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:33:23.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:33:23.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:23 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:33:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.64:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.64 because it doesn't contain any IP SANs" 2024-02-10T07:33:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=warn ts=2024-02-10T07:33:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.136:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.136 because it doesn't contain any IP SANs" 2024-02-10T07:33:24.314 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:23 smithi064 systemd[1]: Stopping Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:24.314 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:23 smithi064 bash[58289]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T07:33:24.314 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 podman[58302]: 2024-02-10 07:33:24.091354896 +0000 UTC m=+0.118722478 container died 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:24.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[24760]: Reconfiguring daemon node-exporter.a on smithi064 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:24.640 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-10T07:33:24.641 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:24 smithi064 ceph-mon[29477]: Reconfiguring daemon node-exporter.a on smithi064 2024-02-10T07:33:24.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-10T07:33:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:24 smithi136 ceph-mon[25575]: Reconfiguring daemon node-exporter.a on smithi064 2024-02-10T07:33:24.923 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 podman[58302]: 2024-02-10 07:33:24.639101011 +0000 UTC m=+0.666468599 container cleanup 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:24.923 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 bash[58302]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T07:33:24.923 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 bash[58329]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 podman[58316]: 2024-02-10 07:33:24.923890426 +0000 UTC m=+0.837059155 container remove 377b0b02dd9f1e4683c63554e57c7d8c86e843d4842ef1eff9bd6feb72ebb0f0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 systemd[1]: Stopped Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 2.622s CPU time. 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:24 smithi064 systemd[1]: Starting Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 podman[58399]: 2024-02-10 07:33:25.054991935 +0000 UTC m=+0.017936234 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T07:33:25.280 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 podman[58399]: 2024-02-10 07:33:25.166336178 +0000 UTC m=+0.129280458 container create 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:25.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:25 smithi136 ceph-mon[25575]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:25.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-mon[24760]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:25.736 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:33:25.737 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-mon[29477]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 podman[58399]: 2024-02-10 07:33:25.402211236 +0000 UTC m=+0.365155532 container init 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 podman[58399]: 2024-02-10 07:33:25.404904589 +0000 UTC m=+0.367848871 container start 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.408Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.408Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.408Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.408Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T07:33:25.738 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T07:33:25.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T07:33:25.740 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T07:33:25.741 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T07:33:25.742 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T07:33:25.743 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a[58413]: ts=2024-02-10T07:33:25.409Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T07:33:25.744 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 bash[58399]: 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 2024-02-10T07:33:25.745 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:33:25 smithi064 systemd[1]: Started Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:26 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:26 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:26 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:26.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:26 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:26 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:26 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:26 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:26 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:26.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:26 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:27 smithi136 ceph-mon[25575]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:33:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:27 smithi136 ceph-mon[25575]: Reconfiguring daemon alertmanager.a on smithi064 2024-02-10T07:33:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:27 smithi136 ceph-mon[25575]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:27.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:27 smithi064 ceph-mon[24760]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:33:27.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:27 smithi064 ceph-mon[24760]: Reconfiguring daemon alertmanager.a on smithi064 2024-02-10T07:33:27.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:27 smithi064 ceph-mon[24760]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:27.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:27 smithi064 ceph-mon[29477]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:33:27.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:27 smithi064 ceph-mon[29477]: Reconfiguring daemon alertmanager.a on smithi064 2024-02-10T07:33:27.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:27 smithi064 ceph-mon[29477]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:28.777 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:33:29.485 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (4m) 20s ago 4m 27.1M - 0.23.0 ba2b418f427c 9e05d1c8d880 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (4m) 19s ago 4m 58.2M - 8.3.5 dad864ee21e9 1d718a66eb18 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (3m) 20s ago 3m 51.2M - 3.5 e1d6a67b021e 517742b65a2b 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283 running (39s) 19s ago 9m 491M - 19.0.0-1361-g290212a1 c328ab9dcfc7 76cb04fd4e2b 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:9283 running (11m) 20s ago 11m 375M - 17.2.0 e1d6a67b021e 49c504cf1f60 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (11m) 20s ago 11m 52.5M 2048M 17.2.0 e1d6a67b021e 0975e381bf03 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (9m) 19s ago 9m 42.3M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (9m) 20s ago 9m 46.0M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 starting - - - - 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (5m) 19s ago 5m 16.8M - 1.3.1 1dbe0e931976 bd193e88eb53 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (9m) 20s ago 9m 54.4M 2736M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (8m) 20s ago 8m 54.9M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (8m) 20s ago 8m 50.4M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:33:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (7m) 20s ago 7m 53.0M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (7m) 19s ago 7m 50.8M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (7m) 19s ago 7m 48.4M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (6m) 19s ago 6m 49.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (6m) 19s ago 6m 52.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (4m) 19s ago 5m 66.7M - 2.33.4 514e6a882f6e 75ee28ab375d 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (4m) 20s ago 4m 71.5M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (3m) 19s ago 3m 71.1M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (3m) 20s ago 3m 70.3M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:33:29.487 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (3m) 19s ago 3m 70.1M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:33:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:29 smithi136 ceph-mon[25575]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:29.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:29 smithi064 ceph-mon[24760]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:29.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:29 smithi064 ceph-mon[29477]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:29.891 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:33:29.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:33:29.891 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T07:33:29.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:33:29.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:33:29.893 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [], 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T07:33:30.231 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "", 2024-02-10T07:33:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:33:30.232 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:33:30.639 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:33:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:30 smithi136 ceph-mon[25575]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:30 smithi136 ceph-mon[25575]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:30 smithi136 ceph-mon[25575]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:30 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2118014704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:33:30.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:30 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[24760]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[24760]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[24760]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2118014704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:33:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[29477]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[29477]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[29477]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2118014704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:33:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:30 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:31.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:31 smithi064 ceph-mon[29477]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:31.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:31 smithi064 ceph-mon[29477]: from='client.25075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:31.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:31 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1858767835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:33:31.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:31 smithi064 ceph-mon[24760]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:31.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:31 smithi064 ceph-mon[24760]: from='client.25075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:31.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:31 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1858767835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:33:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:31 smithi136 ceph-mon[25575]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:31 smithi136 ceph-mon[25575]: from='client.25075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:31 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1858767835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:33:32.314 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:31] "GET /metrics HTTP/1.1" 200 38148 "" "Prometheus/2.33.4" 2024-02-10T07:33:32.722 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.64:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.136:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:33:33.223 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 systemd[1]: Stopping Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:33.223 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 bash[58808]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T07:33:33.507 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:33 smithi064 ceph-mon[24760]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:33.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:33 smithi064 ceph-mon[29477]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:33.508 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[49419]: level=info ts=2024-02-10T07:33:33.222Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:33:33.508 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 podman[58821]: 2024-02-10 07:33:33.243346831 +0000 UTC m=+0.129542310 container died 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:33 smithi136 ceph-mon[25575]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:33.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 podman[58821]: 2024-02-10 07:33:33.605668322 +0000 UTC m=+0.491863803 container cleanup 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:33.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 bash[58821]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T07:33:33.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 bash[58848]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T07:33:34.214 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 podman[58835]: 2024-02-10 07:33:33.873554272 +0000 UTC m=+0.630156087 container remove 9e05d1c8d880da375f14220d9699a7081ca96855fcc2af7b9ab56126126544a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:34.214 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 podman[58835]: 2024-02-10 07:33:33.974334055 +0000 UTC m=+0.730935809 volume remove 63f792d578e4da5efe215ff26d46dc8c3c208c2577428e2db8b84852305e7609 2024-02-10T07:33:34.214 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T07:33:34.214 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 systemd[1]: Stopped Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:34.215 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:33 smithi064 systemd[1]: Starting Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:34.500 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 podman[58917]: 2024-02-10 07:33:34.117711507 +0000 UTC m=+0.019099983 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-10T07:33:34.501 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 podman[58917]: 2024-02-10 07:33:34.216746272 +0000 UTC m=+0.118134749 volume create fa5fdc8c9c9c7342b8b1a8fc4cffa9081d1aa283a91b99e98f5b9af70ebc567e 2024-02-10T07:33:34.501 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 podman[58917]: 2024-02-10 07:33:34.317255348 +0000 UTC m=+0.218643823 container create 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:34.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 podman[58917]: 2024-02-10 07:33:34.586679309 +0000 UTC m=+0.488067786 container init 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:34.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 podman[58917]: 2024-02-10 07:33:34.58927032 +0000 UTC m=+0.490658797 container start 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:34.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.599Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T07:33:34.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.599Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T07:33:34.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.600Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.64 port=9094 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.602Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.627Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.627Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.629Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:34.629Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 bash[58917]: 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc 2024-02-10T07:33:34.815 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:34 smithi064 systemd[1]: Started Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[24760]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:33:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[29477]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:33:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:35 smithi136 ceph-mon[25575]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:35 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:35 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:35 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:33:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:35 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:33:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:35 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:35.993 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:33:36.602 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-mon[24760]: Reconfiguring iscsi.foo.smithi064.yotmxt (dependencies changed)... 2024-02-10T07:33:36.602 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-mon[24760]: Reconfiguring daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:33:36.602 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-mon[24760]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:36.602 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-mon[29477]: Reconfiguring iscsi.foo.smithi064.yotmxt (dependencies changed)... 2024-02-10T07:33:36.602 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-mon[29477]: Reconfiguring daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:33:36.602 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-mon[29477]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:36.871 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:36 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:36.602Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000601s 2024-02-10T07:33:36.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:36 smithi136 ceph-mon[25575]: Reconfiguring iscsi.foo.smithi064.yotmxt (dependencies changed)... 2024-02-10T07:33:36.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:36 smithi136 ceph-mon[25575]: Reconfiguring daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:33:36.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:36 smithi136 ceph-mon[25575]: from='client.34664 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:37.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:37 smithi064 ceph-mon[24760]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:37.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:37 smithi064 ceph-mon[29477]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:37.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:37 smithi136 ceph-mon[25575]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:33:38.200+0000 7f6d51391700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Creating ceph-iscsi config... 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Traceback (most recent call last): 2024-02-10T07:33:38.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return _run_code(code, main_globals, None, 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: exec(code, run_globals) 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:38.474 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Traceback (most recent call last): 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return self.event_loop.get_result(coro, timeout) 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T07:33:38.475 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return future.result(timeout) 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return self.__get_result() 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: raise self._exception 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: config_blobs=daemon_spec.final_config, 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Creating ceph-iscsi config... 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:38.476 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Traceback (most recent call last): 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return _run_code(code, main_globals, None, 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: exec(code, run_globals) 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:38.477 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[24760]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:39.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[24760]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: Creating ceph-iscsi config... 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: exec(code, run_globals) 2024-02-10T07:33:39.815 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T07:33:39.816 INFO:journalctl@ceph.mon.a.smithi064.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: return future.result(timeout) 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.__get_result() 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: raise self._exception 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T07:33:39.817 INFO:journalctl@ceph.mon.a.smithi064.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: Creating ceph-iscsi config... 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.818 INFO:journalctl@ceph.mon.a.smithi064.stdout: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.819 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.819 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:39.819 INFO:journalctl@ceph.mon.a.smithi064.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:33:39.819 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:39.819 INFO:journalctl@ceph.mon.a.smithi064.stdout: exec(code, run_globals) 2024-02-10T07:33:39.819 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[24760]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:33:39.820 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[24760]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[29477]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[29477]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: Creating ceph-iscsi config... 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: exec(code, run_globals) 2024-02-10T07:33:39.821 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.822 INFO:journalctl@ceph.mon.c.smithi064.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: return future.result(timeout) 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T07:33:39.823 INFO:journalctl@ceph.mon.c.smithi064.stdout: return self.__get_result() 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: raise self._exception 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Creating ceph-iscsi config... 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:39.824 INFO:journalctl@ceph.mon.c.smithi064.stdout: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: exec(code, run_globals) 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:39.825 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:39.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:39.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:39.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.826 INFO:journalctl@ceph.mon.c.smithi064.stdout: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[29477]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:33:39.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:39 smithi064 ceph-mon[29477]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:39 smithi136 ceph-mon[25575]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:39 smithi136 ceph-mon[25575]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout: Creating ceph-iscsi config... 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.973 INFO:journalctl@ceph.mon.b.smithi136.stdout: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.974 INFO:journalctl@ceph.mon.b.smithi136.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.974 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:39.974 INFO:journalctl@ceph.mon.b.smithi136.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:33:39.974 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:39.974 INFO:journalctl@ceph.mon.b.smithi136.stdout: exec(code, run_globals) 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T07:33:39.975 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: return future.result(timeout) 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: return self.__get_result() 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: raise self._exception 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: config_blobs=daemon_spec.final_config, 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T07:33:39.976 INFO:journalctl@ceph.mon.b.smithi136.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: Creating ceph-iscsi config... 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: Traceback (most recent call last): 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: exec(code, run_globals) 2024-02-10T07:33:39.977 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:39 smithi136 ceph-mon[25575]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:33:39.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:39 smithi136 ceph-mon[25575]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:33:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:40 smithi064 ceph-mon[24760]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:41.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:40 smithi064 ceph-mon[29477]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:41.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:40 smithi136 ceph-mon[25575]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:42.314 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:33:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:41] "GET /metrics HTTP/1.1" 200 38148 "" "Prometheus/2.33.4" 2024-02-10T07:33:42.722 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:42.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.64:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.136:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:33:43.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:43 smithi064 ceph-mon[24760]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T07:33:43.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:43 smithi064 ceph-mon[29477]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T07:33:43.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:43 smithi136 ceph-mon[25575]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-10T07:33:45.064 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:33:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:33:44.604Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00271104s 2024-02-10T07:33:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:45 smithi064 ceph-mon[24760]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:45.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:45 smithi064 ceph-mon[29477]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:45 smithi136 ceph-mon[25575]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:46.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:33:45 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[24994]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:33:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T07:33:46.521 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:46 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:46.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:46 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:46.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:46 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:47.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:47 smithi136 ceph-mon[25575]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:47 smithi064 ceph-mon[24760]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:47.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:47 smithi064 ceph-mon[29477]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:49.268 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 systemd[1]: Stopping Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:49.268 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 bash[50216]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:33:49.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:49 smithi064 ceph-mon[24760]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T07:33:49.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:49 smithi064 ceph-mon[29477]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T07:33:49.572 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-mon[25575]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T07:33:49.572 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:33:49.572 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T07:33:49.572 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T07:33:49.572 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.268Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.269Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.269Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[42466]: ts=2024-02-10T07:33:49.269Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T07:33:49.573 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 podman[50228]: 2024-02-10 07:33:49.278483743 +0000 UTC m=+0.113746832 container died 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:49.960 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 podman[50228]: 2024-02-10 07:33:49.659756067 +0000 UTC m=+0.495019152 container cleanup 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:49.960 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 bash[50228]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T07:33:49.960 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 bash[50254]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:33:50.219 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 podman[50242]: 2024-02-10 07:33:49.960981027 +0000 UTC m=+0.682993111 container remove 75ee28ab375d3f731006fa58480268860f33bd61e7457d0081860a7dcb76f7f4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:50.220 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T07:33:50.220 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 systemd[1]: Stopped Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:50.220 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.482s CPU time. 2024-02-10T07:33:50.220 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:49 smithi136 systemd[1]: Starting Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:50.220 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 podman[50329]: 2024-02-10 07:33:50.110798005 +0000 UTC m=+0.018953750 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T07:33:50.220 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 podman[50329]: 2024-02-10 07:33:50.220284785 +0000 UTC m=+0.128440514 container create 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 podman[50329]: 2024-02-10 07:33:50.430930175 +0000 UTC m=+0.339085914 container init 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 podman[50329]: 2024-02-10 07:33:50.433322581 +0000 UTC m=+0.341478315 container start 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.456Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.456Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.456Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.456Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi136 (none))" 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.456Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.456Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.458Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T07:33:50.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.459Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T07:33:50.474 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.459Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T07:33:50.474 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.462Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T07:33:50.474 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.462Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.631µs 2024-02-10T07:33:50.474 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.462Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T07:33:50.735 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 bash[50329]: 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c 2024-02-10T07:33:50.735 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 systemd[1]: Started Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:50.735 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.662Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-02-10T07:33:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1902179510' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:33:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2508416569' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1442184411"}]: dispatch 2024-02-10T07:33:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[29477]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T07:33:51.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:51.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1902179510' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:33:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2508416569' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1442184411"}]: dispatch 2024-02-10T07:33:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[24760]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T07:33:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:51.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:50 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:51.115 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1902179510' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:33:51.115 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2508416569' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1442184411"}]: dispatch 2024-02-10T07:33:51.115 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-mon[25575]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-10T07:33:51.115 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:51.115 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:51.115 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.918Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-02-10T07:33:51.115 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.918Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-02-10T07:33:51.115 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.918Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.398µs wal_replay_duration=456.453728ms total_replay_duration=456.518805ms 2024-02-10T07:33:51.116 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.922Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T07:33:51.116 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.922Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T07:33:51.116 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.922Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T07:33:51.116 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.943Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.270884ms db_storage=746ns remote_storage=1.438µs web_handler=336ns query_engine=753ns scrape=396.56µs scrape_sd=72.785µs notify=9.378µs notify_sd=10.605µs rules=20.557293ms 2024-02-10T07:33:51.116 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:33:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:33:50.943Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T07:33:51.468 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 systemd[1]: Stopping Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:51.468 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 bash[50454]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T07:33:51.469 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 podman[50467]: 2024-02-10 07:33:51.287083819 +0000 UTC m=+0.140505353 container died bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:51.722 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 podman[50467]: 2024-02-10 07:33:51.64353149 +0000 UTC m=+0.496953012 container cleanup bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:51.722 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 bash[50467]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T07:33:51.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 bash[50496]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T07:33:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[24760]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-10T07:33:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[24760]: Reconfiguring daemon node-exporter.b on smithi136 2024-02-10T07:33:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2508416569' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1442184411"}]': finished 2024-02-10T07:33:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[24760]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T07:33:52.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4290040445' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/4253215926"}]: dispatch 2024-02-10T07:33:52.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[29477]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-10T07:33:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[29477]: Reconfiguring daemon node-exporter.b on smithi136 2024-02-10T07:33:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2508416569' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1442184411"}]': finished 2024-02-10T07:33:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[29477]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T07:33:52.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:51 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4290040445' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/4253215926"}]: dispatch 2024-02-10T07:33:52.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:51 smithi136 ceph-mon[25575]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-10T07:33:52.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:51 smithi136 ceph-mon[25575]: Reconfiguring daemon node-exporter.b on smithi136 2024-02-10T07:33:52.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:51 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2508416569' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1442184411"}]': finished 2024-02-10T07:33:52.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:51 smithi136 ceph-mon[25575]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T07:33:52.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:51 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4290040445' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/4253215926"}]: dispatch 2024-02-10T07:33:52.174 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 podman[50481]: 2024-02-10 07:33:51.945345296 +0000 UTC m=+0.659326708 container remove bd193e88eb5391705327b313864916ad13ffcd5e783216288f3a682b211a4679 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:52.174 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:33:52.175 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T07:33:52.175 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 systemd[1]: Stopped Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:52.175 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 2.757s CPU time. 2024-02-10T07:33:52.175 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:51 smithi136 systemd[1]: Starting Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:33:52.432 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 podman[50574]: 2024-02-10 07:33:52.076679756 +0000 UTC m=+0.019183995 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-10T07:33:52.432 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 podman[50574]: 2024-02-10 07:33:52.214307293 +0000 UTC m=+0.156811521 container create 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:52.722 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 podman[50574]: 2024-02-10 07:33:52.432661442 +0000 UTC m=+0.375165685 container init 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 podman[50574]: 2024-02-10 07:33:52.435250446 +0000 UTC m=+0.377754674 container start 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.439Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.439Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T07:33:52.723 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T07:33:52.724 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.440Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T07:33:52.725 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T07:33:52.726 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T07:33:52.727 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T07:33:52.727 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T07:33:52.727 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b[50590]: ts=2024-02-10T07:33:52.441Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T07:33:52.727 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 bash[50574]: 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 2024-02-10T07:33:52.727 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:33:52 smithi136 systemd[1]: Started Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4290040445' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/4253215926"}]': finished 2024-02-10T07:33:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T07:33:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/480476905' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]: dispatch 2024-02-10T07:33:53.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]: dispatch 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4290040445' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/4253215926"}]': finished 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/480476905' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]: dispatch 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]: dispatch 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:33:53.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:52 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:53.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4290040445' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/4253215926"}]': finished 2024-02-10T07:33:53.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T07:33:53.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/480476905' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]: dispatch 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]: dispatch 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:33:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:52 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: Reconfiguring grafana.a (dependencies changed)... 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: Reconfiguring daemon grafana.a on smithi136 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]': finished 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2403429704' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]: dispatch 2024-02-10T07:33:54.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]: dispatch 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: Reconfiguring grafana.a (dependencies changed)... 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: Reconfiguring daemon grafana.a on smithi136 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]': finished 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2403429704' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]: dispatch 2024-02-10T07:33:54.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:53 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]: dispatch 2024-02-10T07:33:54.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: Reconfiguring grafana.a (dependencies changed)... 2024-02-10T07:33:54.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T07:33:54.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:54.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: Reconfiguring daemon grafana.a on smithi136 2024-02-10T07:33:54.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/716243094"}]': finished 2024-02-10T07:33:54.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T07:33:54.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2403429704' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]: dispatch 2024-02-10T07:33:54.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:53 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]: dispatch 2024-02-10T07:33:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]': finished 2024-02-10T07:33:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[24760]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T07:33:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1484800839' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1852510286"}]: dispatch 2024-02-10T07:33:55.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[24760]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:55.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]': finished 2024-02-10T07:33:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[29477]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T07:33:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1484800839' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1852510286"}]: dispatch 2024-02-10T07:33:55.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:54 smithi064 ceph-mon[29477]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:55.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:54 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1546573538"}]': finished 2024-02-10T07:33:55.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:54 smithi136 ceph-mon[25575]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T07:33:55.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:54 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1484800839' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1852510286"}]: dispatch 2024-02-10T07:33:55.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:54 smithi136 ceph-mon[25575]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:33:56.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:55 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1484800839' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1852510286"}]': finished 2024-02-10T07:33:56.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:55 smithi064 ceph-mon[24760]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T07:33:56.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:55 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3177606858' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1852510286"}]: dispatch 2024-02-10T07:33:56.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:55 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1484800839' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1852510286"}]': finished 2024-02-10T07:33:56.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:55 smithi064 ceph-mon[29477]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T07:33:56.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:55 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3177606858' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1852510286"}]: dispatch 2024-02-10T07:33:56.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:55 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1484800839' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/1852510286"}]': finished 2024-02-10T07:33:56.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:55 smithi136 ceph-mon[25575]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T07:33:56.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:55 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3177606858' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1852510286"}]: dispatch 2024-02-10T07:33:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3177606858' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1852510286"}]': finished 2024-02-10T07:33:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[24760]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T07:33:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3854603896' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]: dispatch 2024-02-10T07:33:57.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]: dispatch 2024-02-10T07:33:57.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[24760]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:57.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3177606858' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1852510286"}]': finished 2024-02-10T07:33:57.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[29477]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T07:33:57.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3854603896' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]: dispatch 2024-02-10T07:33:57.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]: dispatch 2024-02-10T07:33:57.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:56 smithi064 ceph-mon[29477]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:56 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3177606858' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/1852510286"}]': finished 2024-02-10T07:33:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:56 smithi136 ceph-mon[25575]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T07:33:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:56 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3854603896' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]: dispatch 2024-02-10T07:33:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:56 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]: dispatch 2024-02-10T07:33:57.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:56 smithi136 ceph-mon[25575]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:58.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:57 smithi064 ceph-mon[24760]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]': finished 2024-02-10T07:33:58.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:57 smithi064 ceph-mon[24760]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T07:33:58.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:57 smithi064 ceph-mon[29477]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]': finished 2024-02-10T07:33:58.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:57 smithi064 ceph-mon[29477]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T07:33:58.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:57 smithi136 ceph-mon[25575]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3476907811"}]': finished 2024-02-10T07:33:58.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:57 smithi136 ceph-mon[25575]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:33:58 smithi064 ceph-mon[24760]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:33:59.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:33:58 smithi064 ceph-mon[29477]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:33:59.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:33:58 smithi136 ceph-mon[25575]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:01.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:00 smithi064 ceph-mon[24760]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:01.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:00 smithi064 ceph-mon[24760]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:01.006 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:00 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:01.006 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:00 smithi064 ceph-mon[29477]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:01.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:00 smithi064 ceph-mon[29477]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:01.007 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:00 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:01.007 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:34:01.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:00 smithi136 ceph-mon[25575]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:01.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:00 smithi136 ceph-mon[25575]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:01.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:00 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:01.666 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:34:01.666 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 starting - - - - 2024-02-10T07:34:01.666 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (5m) 52s ago 5m 58.2M - 8.3.5 dad864ee21e9 1d718a66eb18 2024-02-10T07:34:01.666 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (4m) 52s ago 4m 51.2M - 3.5 e1d6a67b021e 517742b65a2b 2024-02-10T07:34:01.666 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283 running (71s) 52s ago 9m 491M - 19.0.0-1361-g290212a1 c328ab9dcfc7 76cb04fd4e2b 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:9283 running (11m) 52s ago 11m 375M - 17.2.0 e1d6a67b021e 49c504cf1f60 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (11m) 52s ago 11m 52.5M 2048M 17.2.0 e1d6a67b021e 0975e381bf03 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (10m) 52s ago 10m 42.3M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (10m) 52s ago 10m 46.0M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 starting - - - - 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 starting - - - - 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (9m) 52s ago 9m 54.4M 2736M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (9m) 52s ago 9m 54.9M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (8m) 52s ago 8m 50.4M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (8m) 52s ago 8m 53.0M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (7m) 52s ago 7m 50.8M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (7m) 52s ago 7m 48.4M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (7m) 52s ago 7m 49.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (6m) 52s ago 6m 52.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 starting - - - - 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (4m) 52s ago 4m 71.5M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (4m) 52s ago 4m 71.1M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (4m) 52s ago 4m 70.3M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:34:01.668 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (4m) 52s ago 4m 70.1M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:34:02.036 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:01 smithi136 ceph-mon[25575]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:02.037 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:01 smithi136 ceph-mon[25575]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:02.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:01 smithi064 ceph-mon[24760]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:02.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:01 smithi064 ceph-mon[24760]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:02.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:01 smithi064 ceph-mon[29477]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:02.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:01 smithi064 ceph-mon[29477]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:02.069 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:34:02.069 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:34:02.069 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T07:34:02.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:34:02.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:34:02.071 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [], 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "", 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:34:02.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:34:02] "GET /metrics HTTP/1.1" 200 38146 "" "Prometheus/2.33.4" 2024-02-10T07:34:02.817 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:34:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[24760]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/4276487483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[24760]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:34:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[24760]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[29477]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/4276487483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[29477]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:34:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:02 smithi064 ceph-mon[29477]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:03.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:02 smithi136 ceph-mon[25575]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:03.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:02 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/4276487483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:02 smithi136 ceph-mon[25575]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:34:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:02 smithi136 ceph-mon[25575]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:04.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:03 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2974352364' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:34:04.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:03 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2974352364' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:34:04.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:03 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2974352364' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:34:05.032 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:04 smithi136 ceph-mon[25575]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:05.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:04 smithi064 ceph-mon[24760]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:05.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:04 smithi064 ceph-mon[29477]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:06.294 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 systemd[1]: Stopping Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:06.722 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 bash[50788]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T07:34:06.723 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[41675]: t=2024-02-10T07:34:06+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T07:34:06.723 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 podman[50802]: 2024-02-10 07:34:06.425243083 +0000 UTC m=+0.121252783 container died 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, description=Ceph Grafana Container, name=ubi8, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vendor=Red Hat, Inc., version=8.5) 2024-02-10T07:34:07.049 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 podman[50802]: 2024-02-10 07:34:06.814485403 +0000 UTC m=+0.510495113 container cleanup 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.buildah.version=1.24.2, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, vendor=Red Hat, Inc., name=ubi8, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, release=236.1648460182, version=8.5) 2024-02-10T07:34:07.049 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 bash[50802]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T07:34:07.049 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:06 smithi136 bash[50830]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T07:34:07.405 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-mon[25575]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 podman[50816]: 2024-02-10 07:34:07.049296334 +0000 UTC m=+0.623719160 container remove 1d718a66eb1866b8a60f6c91941fd61094bbc4746759a35a2de36c6ee14fbdd4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, description=Ceph Grafana Container, release=236.1648460182, io.openshift.tags=base rhel8, distribution-scope=public, maintainer=Paul Cuzner , version=8.5, name=ubi8, vcs-type=git, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 systemd[1]: Stopped Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Consumed 2.577s CPU time. 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 systemd[1]: Starting Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 podman[50896]: 2024-02-10 07:34:07.181834613 +0000 UTC m=+0.017640938 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-10T07:34:07.406 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 podman[50896]: 2024-02-10 07:34:07.292127181 +0000 UTC m=+0.127933505 container create c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vcs-type=git, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , distribution-scope=public, architecture=x86_64, io.openshift.expose-services=, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.openshift.tags=base rhel8) 2024-02-10T07:34:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:07 smithi064 ceph-mon[24760]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:34:07.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:07 smithi064 ceph-mon[29477]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:34:07.853 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STOPPING 2024-02-10T07:34:07.853 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:07.853 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STOPPED 2024-02-10T07:34:07.853 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STARTING 2024-02-10T07:34:07.853 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Serving on http://:::9283 2024-02-10T07:34:07.853 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STARTED 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 podman[50896]: 2024-02-10 07:34:07.52797211 +0000 UTC m=+0.363778434 container init c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, release=236.1648460182, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2) 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 podman[50896]: 2024-02-10 07:34:07.530802364 +0000 UTC m=+0.366608690 container start c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, release=236.1648460182, version=8.5) 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T07:34:07.854 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=526.976µs 2024-02-10T07:34:07.855 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 bash[50896]: c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 systemd[1]: Started Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-10T07:34:07.856 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:34:07+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STOPPING 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STOPPED 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STARTING 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Serving on http://:::9283 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STARTED 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STOPPING 2024-02-10T07:34:08.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:08.224 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STOPPED 2024-02-10T07:34:08.224 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:07] ENGINE Bus STARTING 2024-02-10T07:34:08.224 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:08] ENGINE Serving on http://:::9283 2024-02-10T07:34:08.224 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:08] ENGINE Bus STARTED 2024-02-10T07:34:08.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:08.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:34:08.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:08.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:08.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:08.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:08.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi136.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi136.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi136.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi136.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:08.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:08.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:08.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T07:34:08.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T07:34:08.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:08 smithi136 ceph-mon[25575]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-02-10T07:34:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi136.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi136.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:09.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi136.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi136.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[24760]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:09.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi064.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi136.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi136.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi136.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi136.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:09.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:09.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:09.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T07:34:09.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-10T07:34:09.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:08 smithi064 ceph-mon[29477]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: Upgrade: Updating mgr.y 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:10.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[24760]: Deploying daemon mgr.y on smithi064 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: Upgrade: Updating mgr.y 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:10.699 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:10 smithi064 ceph-mon[29477]: Deploying daemon mgr.y on smithi064 2024-02-10T07:34:10.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: Upgrade: Updating mgr.y 2024-02-10T07:34:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:34:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:10 smithi136 ceph-mon[25575]: Deploying daemon mgr.y on smithi064 2024-02-10T07:34:11.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:11 smithi064 systemd[1]: Stopping Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:11.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:11 smithi136 ceph-mon[25575]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:11.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:11 smithi136 ceph-mon[25575]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:11 smithi064 ceph-mon[24760]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:11.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:11 smithi064 ceph-mon[24760]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:11.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:11 smithi064 ceph-mon[29477]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:11.803 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:11 smithi064 ceph-mon[29477]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:11.803 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:11 smithi064 podman[60021]: 2024-02-10 07:34:11.524772483 +0000 UTC m=+0.149373788 container died 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, name=centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.tags=base centos centos-stream, release=754, architecture=x86_64) 2024-02-10T07:34:12.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:11 smithi064 podman[60021]: 2024-02-10 07:34:11.876946053 +0000 UTC m=+0.501547429 container cleanup 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, io.k8s.display-name=CentOS Stream 8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_CLEAN=True, version=8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T07:34:12.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:11 smithi064 bash[60021]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y 2024-02-10T07:34:12.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:34:12] "GET /metrics HTTP/1.1" 200 38146 "" "Prometheus/2.33.4" 2024-02-10T07:34:12.506 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 podman[60035]: 2024-02-10 07:34:12.153475231 +0000 UTC m=+0.629438067 container remove 49c504cf1f6070114e7c8d17f2f4e1513996c3dee1e2e2d5034c14a86528a5a9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, io.buildah.version=1.19.8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-type=git, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, maintainer=Guillaume Abrioux , ceph=True) 2024-02-10T07:34:12.506 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:34:12.506 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-02-10T07:34:12.506 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 systemd[1]: Stopped Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:12.506 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service: Consumed 56.162s CPU time. 2024-02-10T07:34:12.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 systemd[1]: Starting Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:12.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 podman[60165]: 2024-02-10 07:34:12.620528734 +0000 UTC m=+0.017830417 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:34:12.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 podman[60165]: 2024-02-10 07:34:12.737886483 +0000 UTC m=+0.135188157 container create febd4e49df031318ce3519a4febb48e750c4c599871191fcbde03610c5dd5f50 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T07:34:13.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 podman[60165]: 2024-02-10 07:34:12.94867072 +0000 UTC m=+0.345972396 container init febd4e49df031318ce3519a4febb48e750c4c599871191fcbde03610c5dd5f50 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T07:34:13.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 podman[60165]: 2024-02-10 07:34:12.951326387 +0000 UTC m=+0.348628070 container start febd4e49df031318ce3519a4febb48e750c4c599871191fcbde03610c5dd5f50 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, RELEASE=main-290212a, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-10T07:34:13.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:12 smithi064 ceph-mgr[60184]: -- 172.21.15.64:0/902194567 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f8d1b4c1a0 con 0x55f8d1b22c00 2024-02-10T07:34:13.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:13 smithi064 bash[60165]: febd4e49df031318ce3519a4febb48e750c4c599871191fcbde03610c5dd5f50 2024-02-10T07:34:13.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:13 smithi064 systemd[1]: Started Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:13.631 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-mon[24760]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:13.632 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:13.632 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:13.632 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:13.530+0000 7f7e7d8ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:34:13.632 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-mon[29477]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:13.632 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:13.632 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:13.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:13 smithi136 ceph-mon[25575]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:13.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:13 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:13.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:13 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.594 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:13.630+0000 7f7e7d8ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:34:14.595 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:14.338+0000 7f7e7d8ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:34:14.595 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:14.433+0000 7f7e7d8ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:34:14.595 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:14.583+0000 7f7e7d8ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:34:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:14 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:14 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:14 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:14 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:14 smithi136 ceph-mon[25575]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[24760]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:14.686+0000 7f7e7d8ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:34:14.985 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:14.775+0000 7f7e7d8ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:34:14.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:14.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:14 smithi064 ceph-mon[29477]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:15.274 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:15.060+0000 7f7e7d8ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:34:15.274 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:15.179+0000 7f7e7d8ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:34:15.562 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:15.273+0000 7f7e7d8ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:34:15.562 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:15.379+0000 7f7e7d8ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:34:15.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:15.815 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:15.561+0000 7f7e7d8ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:34:15.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:15 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:16.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:15 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:16.955 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:16 smithi064 ceph-mon[24760]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:16.955 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:16 smithi064 ceph-mon[29477]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:16.955 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:16 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:16.660+0000 7f7e7d8ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:34:16.955 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:16 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:16.797+0000 7f7e7d8ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:34:16.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:16 smithi136 ceph-mon[25575]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:17.244 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:17.149+0000 7f7e7d8ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:34:17.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:17.244+0000 7f7e7d8ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:34:17.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:17.463+0000 7f7e7d8ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:34:17.963 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:17.660+0000 7f7e7d8ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:34:18.236 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:17.963+0000 7f7e7d8ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:34:18.237 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:18 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:18.235+0000 7f7e7d8ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:34:18.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:18 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:18.490+0000 7f7e7d8ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:34:18.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:34:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-mon[24760]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:19.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:19.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:19.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-mon[29477]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:19 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:19.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:19 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:19.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:19 smithi136 ceph-mon[25575]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:19.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:19.364+0000 7f7e7d8ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:34:20.160 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:19 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:19.978+0000 7f7e7d8ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:34:20.160 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:20.084+0000 7f7e7d8ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:34:20.435 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:20.435 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:20.436 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:20.436 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:20.436 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:20.314+0000 7f7e7d8ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:34:20.436 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:20] ENGINE Bus STARTING 2024-02-10T07:34:20.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:20 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:20.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:20 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:20.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: CherryPy Checker: 2024-02-10T07:34:20.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: The Application mounted at '' has an empty config. 2024-02-10T07:34:20.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:34:20.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:20] ENGINE Serving on http://:::9283 2024-02-10T07:34:20.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:20] ENGINE Bus STARTED 2024-02-10T07:34:21.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:21.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:21.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: Standby manager daemon y restarted 2024-02-10T07:34:21.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: Standby manager daemon y started 2024-02-10T07:34:21.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T07:34:21.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:21.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T07:34:21.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:21 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: Standby manager daemon y restarted 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: Standby manager daemon y started 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: from='client.15195 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: Standby manager daemon y restarted 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: Standby manager daemon y started 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-10T07:34:21.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:21 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:22.203 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:22 smithi136 ceph-mon[25575]: mgrmap e27: x(active, since 82s), standbys: y 2024-02-10T07:34:22.203 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:34:22] "GET /metrics HTTP/1.1" 200 38144 "" "Prometheus/2.33.4" 2024-02-10T07:34:22.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:22 smithi064 ceph-mon[24760]: mgrmap e27: x(active, since 82s), standbys: y 2024-02-10T07:34:22.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:22 smithi064 ceph-mon[29477]: mgrmap e27: x(active, since 82s), standbys: y 2024-02-10T07:34:23.451 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:23 smithi136 ceph-mon[25575]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:23.451 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:23.451 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:23 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:23.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:23 smithi064 ceph-mon[29477]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:23.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:23.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:23 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:23 smithi064 ceph-mon[24760]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:23.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:23 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:34:25.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:24 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:25.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:25.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:25.276 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' 2024-02-10T07:34:25.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:34:25.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:34:25.278 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:24 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:26.186 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:25 smithi064 ceph-mon[24760]: Reconfiguring iscsi.foo.smithi064.yotmxt (dependencies changed)... 2024-02-10T07:34:26.186 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:25 smithi064 ceph-mon[24760]: Reconfiguring daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:34:26.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:25 smithi064 ceph-mon[29477]: Reconfiguring iscsi.foo.smithi064.yotmxt (dependencies changed)... 2024-02-10T07:34:26.187 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:25 smithi064 ceph-mon[29477]: Reconfiguring daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:34:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:25 smithi136 ceph-mon[25575]: Reconfiguring iscsi.foo.smithi064.yotmxt (dependencies changed)... 2024-02-10T07:34:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:25 smithi136 ceph-mon[25575]: Reconfiguring daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:34:27.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:26 smithi064 ceph-mon[24760]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:27.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:26 smithi064 ceph-mon[29477]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:27.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:26 smithi136 ceph-mon[25575]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:28.131 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-mon[25575]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T07:34:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T07:34:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-mon[25575]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:27.816+0000 7f6d51391700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Creating ceph-iscsi config... 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Traceback (most recent call last): 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:34:28.133 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return _run_code(code, main_globals, None, 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: exec(code, run_globals) 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:34:28.134 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:34:28.135 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:34:28.135 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:34:28.135 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Traceback (most recent call last): 2024-02-10T07:34:28.135 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-10T07:34:28.135 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-10T07:34:28.135 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2368, in _daemon_action 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return self.event_loop.get_result(coro, timeout) 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return future.result(timeout) 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-10T07:34:28.136 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return self.__get_result() 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: raise self._exception 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: config_blobs=daemon_spec.final_config, 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-10T07:34:28.137 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:34:28.138 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Creating ceph-iscsi config... 2024-02-10T07:34:28.138 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:34:28.138 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:34:28.138 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Non-zero exit code 1 from systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:34:28.138 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: Traceback (most recent call last): 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: return _run_code(code, main_globals, None, 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: exec(code, run_globals) 2024-02-10T07:34:28.139 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1065, in deploy_daemon 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:34:28.140 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: RuntimeError: Failed command: systemctl restart ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:34:28.141 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:34:28.141 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:27.985+0000 7f6dbecda700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-10T07:34:28.141 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ignoring --setuser ceph since I am not root 2024-02-10T07:34:28.141 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: ignoring --setgroup ceph since I am not root 2024-02-10T07:34:28.141 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mgr[46731]: -- 172.21.15.136:0/2574438048 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557ede4d01a0 con 0x557ede4a7000 2024-02-10T07:34:28.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:28.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[24760]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T07:34:28.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T07:34:28.240 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[24760]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[29477]: from='mgr.34838 172.21.15.136:0/1509060880' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-mon[29477]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:27] ENGINE Bus STOPPING 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:28] ENGINE Bus STOPPED 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:28] ENGINE Bus STARTING 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:28] ENGINE Serving on http://:::9283 2024-02-10T07:34:28.241 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:34:28] ENGINE Bus STARTED 2024-02-10T07:34:28.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:28.568+0000 7f569be2f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:34:28.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:28.665+0000 7f569be2f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:34:28.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:34:29.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.34838 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: mgrmap e28: y(active, starting, since 0.15289s) 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:34:29.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: Manager daemon y is now available 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:34:29.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.34838 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: mgrmap e28: y(active, starting, since 0.15289s) 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:34:29.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: Manager daemon y is now available 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:34:29.318 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:28 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:34:29.325 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.34838 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-10T07:34:29.325 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: mgrmap e28: y(active, starting, since 0.15289s) 2024-02-10T07:34:29.325 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:34:29.325 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:34:29.325 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:34:29.325 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: Manager daemon y is now available 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:29.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:34:29.327 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:34:29.327 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:34:29.327 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:28 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:34:29.598 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:29.359+0000 7f569be2f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:34:29.598 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:29.453+0000 7f569be2f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:34:29.962 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:29.597+0000 7f569be2f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:34:29.962 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:29.698+0000 7f569be2f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:34:29.962 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:29.791+0000 7f569be2f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:34:30.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:30.060+0000 7f569be2f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:34:30.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:30.177+0000 7f569be2f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:34:30.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-mon[25575]: mgrmap e29: y(active, since 1.16743s) 2024-02-10T07:34:30.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-mon[25575]: [10/Feb/2024:07:34:29] ENGINE Bus STARTING 2024-02-10T07:34:30.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-mon[25575]: [10/Feb/2024:07:34:29] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T07:34:30.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-mon[25575]: [10/Feb/2024:07:34:29] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:34:30.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-mon[25575]: [10/Feb/2024:07:34:29] ENGINE Bus STARTED 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[24760]: mgrmap e29: y(active, since 1.16743s) 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[24760]: [10/Feb/2024:07:34:29] ENGINE Bus STARTING 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[24760]: [10/Feb/2024:07:34:29] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[24760]: [10/Feb/2024:07:34:29] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[24760]: [10/Feb/2024:07:34:29] ENGINE Bus STARTED 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[29477]: mgrmap e29: y(active, since 1.16743s) 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[29477]: [10/Feb/2024:07:34:29] ENGINE Bus STARTING 2024-02-10T07:34:30.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[29477]: [10/Feb/2024:07:34:29] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T07:34:30.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[29477]: [10/Feb/2024:07:34:29] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:34:30.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:30 smithi064 ceph-mon[29477]: [10/Feb/2024:07:34:29] ENGINE Bus STARTED 2024-02-10T07:34:30.556 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:30.266+0000 7f569be2f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:34:30.556 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:30.361+0000 7f569be2f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:34:30.556 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:30.555+0000 7f569be2f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:34:31.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:31 smithi136 ceph-mon[25575]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:31.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:31 smithi064 ceph-mon[24760]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:31.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:31 smithi064 ceph-mon[29477]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:31.843 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:31.571+0000 7f569be2f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:34:31.844 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:31 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:31.707+0000 7f569be2f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:34:32.163 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:32 smithi136 ceph-mon[25575]: mgrmap e30: y(active, since 3s) 2024-02-10T07:34:32.163 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:32.056+0000 7f569be2f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:34:32.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:32 smithi064 ceph-mon[29477]: mgrmap e30: y(active, since 3s) 2024-02-10T07:34:32.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:32 smithi064 ceph-mon[24760]: mgrmap e30: y(active, since 3s) 2024-02-10T07:34:32.433 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:32.162+0000 7f569be2f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:34:32.433 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:32.368+0000 7f569be2f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:34:32.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:32.561+0000 7f569be2f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:34:33.143 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:32.863+0000 7f569be2f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:34:33.179 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:34:33.397 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:33.142+0000 7f569be2f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:34:33.397 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:33.396+0000 7f569be2f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:34:33.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-mon[25575]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:33.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:33 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[24760]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[29477]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:33.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:33 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:33.878 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:34:33.878 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (59s) 1s ago 5m 14.2M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (26s) 0s ago 5m 45.4M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 unknown 1s ago 4m - - 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283 running (103s) 0s ago 10m 74.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 76cb04fd4e2b 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (21s) 1s ago 12m 472M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (12m) 1s ago 12m 53.5M 2048M 17.2.0 e1d6a67b021e 0975e381bf03 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (10m) 0s ago 10m 43.4M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (10m) 1s ago 10m 47.6M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (68s) 1s ago 6m 11.4M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (41s) 0s ago 6m 8082k - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (10m) 1s ago 10m 54.5M 2736M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (9m) 1s ago 9m 55.7M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (9m) 1s ago 9m 50.1M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (8m) 1s ago 8m 54.2M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (8m) 0s ago 8m 51.8M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:34:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (8m) 0s ago 8m 48.7M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (7m) 0s ago 7m 50.4M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (7m) 0s ago 7m 53.3M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (43s) 0s ago 6m 53.6M - 2.33.4 514e6a882f6e 2200ccb71486 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (5m) 1s ago 5m 72.2M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (5m) 0s ago 5m 71.6M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (5m) 1s ago 4m 70.9M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:34:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (4m) 0s ago 4m 70.8M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:34:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:34:34.296 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:34:34.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:34 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.632 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:34:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:34:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:34:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "", 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:34:34.633 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:34:34.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:34.244+0000 7f569be2f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:34:34.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:34.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:34.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:35.044 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:34:35.156 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:34.831+0000 7f569be2f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:34:35.156 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:34 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:34.933+0000 7f569be2f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:34:35.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:35.155+0000 7f569be2f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:34:35.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:35] ENGINE Bus STARTING 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: CherryPy Checker: 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: The Application mounted at '' has an empty config. 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:35] ENGINE Serving on http://:::9283 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[46727]: [10/Feb/2024:07:34:35] ENGINE Bus STARTED 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/2930754668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3127703334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: Standby manager daemon x started 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:34:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:35.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:34:35.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:35 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/2930754668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3127703334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:34:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: Standby manager daemon x started 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/2930754668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/3127703334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: Standby manager daemon x started 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:34:35.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:35 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2998128959' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:37.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:36 smithi064 ceph-mon[29477]: mgrmap e31: y(active, since 7s), standbys: x 2024-02-10T07:34:37.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:36 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:34:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:36 smithi064 ceph-mon[24760]: mgrmap e31: y(active, since 7s), standbys: x 2024-02-10T07:34:37.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:36 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:34:37.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:36 smithi136 ceph-mon[25575]: mgrmap e31: y(active, since 7s), standbys: x 2024-02-10T07:34:37.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:36 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:34:38.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-10T07:34:38.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.055 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.056 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:37 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:37 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:34:38.863 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:38 smithi136 ceph-mon[25575]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:34:39.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[24760]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:34:39.059 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:38 smithi064 ceph-mon[29477]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:39 smithi136 ceph-mon[25575]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:39 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1552648151' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:34:40.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:39 smithi064 ceph-mon[24760]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:34:40.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:39 smithi064 ceph-mon[24760]: from='client.? 172.21.15.64:0/1552648151' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:34:40.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:39 smithi064 ceph-mon[29477]: Reconfiguring daemon prometheus.a on smithi136 2024-02-10T07:34:40.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:39 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1552648151' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:34:41.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:40 smithi064 ceph-mon[24760]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:34:41.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:40 smithi064 ceph-mon[29477]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:34:41.180 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:40 smithi136 ceph-mon[25575]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:34:41.464 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 systemd[1]: Stopping Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:41.465 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 bash[53655]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:34:41.722 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.464Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[50345]: ts=2024-02-10T07:34:41.465Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T07:34:41.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 podman[53668]: 2024-02-10 07:34:41.474885168 +0000 UTC m=+0.113966235 container died 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.066 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 podman[53668]: 2024-02-10 07:34:41.881077411 +0000 UTC m=+0.520158479 container cleanup 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.066 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:41 smithi136 bash[53668]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T07:34:42.316 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 podman[53682]: 2024-02-10 07:34:42.065853543 +0000 UTC m=+0.592533380 container remove 2200ccb714861988ca7f45f97a9cf542a6011381621c0caa1b9297d78d130a0c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.316 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 bash[53697]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:34:42.316 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T07:34:42.317 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 systemd[1]: Stopped Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:42.317 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 systemd[1]: Starting Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 podman[53766]: 2024-02-10 07:34:42.220018608 +0000 UTC m=+0.018661400 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 podman[53766]: 2024-02-10 07:34:42.325176147 +0000 UTC m=+0.123818940 container create 82bac0a679e8e2782628b270a6c311f2bbad1ea677c733ed01b448ec3711126c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 podman[53766]: 2024-02-10 07:34:42.535582448 +0000 UTC m=+0.334225244 container init 82bac0a679e8e2782628b270a6c311f2bbad1ea677c733ed01b448ec3711126c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 podman[53766]: 2024-02-10 07:34:42.538611038 +0000 UTC m=+0.337253832 container start 82bac0a679e8e2782628b270a6c311f2bbad1ea677c733ed01b448ec3711126c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.561Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.561Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.561Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.561Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi136 (none))" 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.561Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.561Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T07:34:42.567 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.563Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T07:34:42.568 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.564Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T07:34:42.568 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.565Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T07:34:42.568 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.567Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T07:34:42.568 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.567Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.61µs 2024-02-10T07:34:42.568 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.567Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T07:34:42.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-mon[25575]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:34:42.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:42.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:42.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 bash[53766]: 82bac0a679e8e2782628b270a6c311f2bbad1ea677c733ed01b448ec3711126c 2024-02-10T07:34:42.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 systemd[1]: Started Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:42.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:42.774Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-02-10T07:34:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[24760]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:34:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:43.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:43.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[29477]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:34:43.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:43.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:43.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:43.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:42 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.018Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.059Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.059Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.060Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.585µs wal_replay_duration=492.786873ms total_replay_duration=492.838473ms 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.063Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.063Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.063Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T07:34:43.325 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.082Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.590158ms db_storage=767ns remote_storage=1.433µs web_handler=359ns query_engine=763ns scrape=638.216µs scrape_sd=90.403µs notify=9.936µs notify_sd=8.556µs rules=18.507405ms 2024-02-10T07:34:43.326 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:43.083Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T07:34:43.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-mon[25575]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:43.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:43.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:43 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:44.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:43 smithi064 ceph-mon[29477]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:44.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:43 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:44.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:43 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:44.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:43 smithi064 ceph-mon[24760]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:44.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:43 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:44.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:43 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:44 smithi136 ceph-mon[25575]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:34:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:44 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:44 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:44 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:34:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:44 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[24760]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:34:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:34:45.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[29477]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:34:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:34:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:34:45.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:44 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:46.051 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:45 smithi136 ceph-mon[25575]: Upgrade: Updating mgr.x 2024-02-10T07:34:46.051 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:45 smithi136 ceph-mon[25575]: Deploying daemon mgr.x on smithi136 2024-02-10T07:34:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:45 smithi064 ceph-mon[24760]: Upgrade: Updating mgr.x 2024-02-10T07:34:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:45 smithi064 ceph-mon[24760]: Deploying daemon mgr.x on smithi136 2024-02-10T07:34:46.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:45 smithi064 ceph-mon[29477]: Upgrade: Updating mgr.x 2024-02-10T07:34:46.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:45 smithi064 ceph-mon[29477]: Deploying daemon mgr.x on smithi136 2024-02-10T07:34:46.620 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:46 smithi136 systemd[1]: Stopping Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:46.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:46 smithi136 ceph-mon[25575]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:34:46.899 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:46 smithi136 podman[54121]: 2024-02-10 07:34:46.620089218 +0000 UTC m=+0.137197716 container died 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T07:34:47.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:46 smithi064 ceph-mon[24760]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:34:47.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:46 smithi064 ceph-mon[29477]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:34:47.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:46 smithi136 podman[54121]: 2024-02-10 07:34:46.981318711 +0000 UTC m=+0.498427209 container cleanup 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a) 2024-02-10T07:34:47.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:46 smithi136 bash[54121]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x 2024-02-10T07:34:47.597 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 podman[54135]: 2024-02-10 07:34:47.266009431 +0000 UTC m=+0.644214417 container remove 76cb04fd4e2b10ef4f919ea592c34f4854d523dfa6a8da4f7535e933be7bd0f6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, RELEASE=main-290212a) 2024-02-10T07:34:47.598 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:34:47.598 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T07:34:47.598 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 systemd[1]: Stopped Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:47.598 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Consumed 27.049s CPU time. 2024-02-10T07:34:47.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 systemd[1]: Starting Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:47.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 podman[54266]: 2024-02-10 07:34:47.715801494 +0000 UTC m=+0.018436979 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:34:47.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:47 smithi136 podman[54266]: 2024-02-10 07:34:47.842027161 +0000 UTC m=+0.144662646 container create 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:34:48.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 podman[54266]: 2024-02-10 07:34:48.086652443 +0000 UTC m=+0.389287928 container init 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-10T07:34:48.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 podman[54266]: 2024-02-10 07:34:48.089265529 +0000 UTC m=+0.391901014 container start 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:34:48.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 ceph-mgr[54285]: -- 172.21.15.136:0/3531910311 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5566195081a0 con 0x5566194dec00 2024-02-10T07:34:48.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 bash[54266]: 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc 2024-02-10T07:34:48.473 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 systemd[1]: Started Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:48.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:48 smithi136 ceph-mon[25575]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 436 B/s rd, 0 op/s 2024-02-10T07:34:48.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:48 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:48.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:48 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:48.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:48.651+0000 7f90f0ada200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:34:48.973 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:48.755+0000 7f90f0ada200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:34:49.078 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:48 smithi064 ceph-mon[29477]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 436 B/s rd, 0 op/s 2024-02-10T07:34:49.079 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:48 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:49.079 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:48 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:49.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:48 smithi064 ceph-mon[24760]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 436 B/s rd, 0 op/s 2024-02-10T07:34:49.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:48 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:49.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:48 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:49.691 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:49.438+0000 7f90f0ada200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:34:49.691 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:49.531+0000 7f90f0ada200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:34:49.952 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:49.689+0000 7f90f0ada200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:34:49.952 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:49.786+0000 7f90f0ada200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:34:49.952 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:49 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:49.870+0000 7f90f0ada200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:34:50.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:50.144+0000 7f90f0ada200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:34:50.650 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:50.255+0000 7f90f0ada200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:34:50.650 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:50.341+0000 7f90f0ada200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:34:50.650 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:50.447+0000 7f90f0ada200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:34:50.650 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:50.649+0000 7f90f0ada200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:34:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-mon[25575]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:50.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:50 smithi136 ceph-mon[25575]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:50 smithi064 ceph-mon[29477]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:50 smithi064 ceph-mon[29477]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:51.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:50 smithi064 ceph-mon[24760]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:51.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:50 smithi064 ceph-mon[24760]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:51.868 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:51.658+0000 7f90f0ada200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:34:51.868 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:51 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:51.792+0000 7f90f0ada200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:34:52.450 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:52.144+0000 7f90f0ada200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:34:52.451 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:52.236+0000 7f90f0ada200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:34:52.451 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:52.450+0000 7f90f0ada200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:34:52.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:52.644+0000 7f90f0ada200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:34:53.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:52 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:52.960+0000 7f90f0ada200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:34:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:52 smithi136 ceph-mon[25575]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:52 smithi064 ceph-mon[24760]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:52 smithi064 ceph-mon[29477]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:53.494 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:53.238+0000 7f90f0ada200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:34:53.905 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:53.494+0000 7f90f0ada200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:34:54.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:53 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:53 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:53 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:53 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:53 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:53 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.690 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:54.361+0000 7f90f0ada200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:34:54.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:54 smithi136 ceph-mon[25575]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:54.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:54 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:54.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:54 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:55.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:54.973+0000 7f90f0ada200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:34:55.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:55.077+0000 7f90f0ada200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:34:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:54 smithi064 ceph-mon[24760]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:54 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:55.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:54 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:54 smithi064 ceph-mon[29477]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:54 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:55.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:54 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:55.722 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:55.297+0000 7f90f0ada200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:34:55.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: [10/Feb/2024:07:34:55] ENGINE Bus STARTING 2024-02-10T07:34:55.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: CherryPy Checker: 2024-02-10T07:34:55.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: The Application mounted at '' has an empty config. 2024-02-10T07:34:55.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:34:55.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: [10/Feb/2024:07:34:55] ENGINE Serving on http://:::9283 2024-02-10T07:34:55.724 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: [10/Feb/2024:07:34:55] ENGINE Bus STARTED 2024-02-10T07:34:56.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-mon[25575]: Standby manager daemon x restarted 2024-02-10T07:34:56.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-mon[25575]: Standby manager daemon x started 2024-02-10T07:34:56.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:34:56.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:56.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:34:56.093 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:55 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:56.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[24760]: Standby manager daemon x restarted 2024-02-10T07:34:56.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[24760]: Standby manager daemon x started 2024-02-10T07:34:56.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:34:56.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:56.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:34:56.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[24760]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:56.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[29477]: Standby manager daemon x restarted 2024-02-10T07:34:56.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[29477]: Standby manager daemon x started 2024-02-10T07:34:56.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:34:56.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:34:56.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:34:56.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:55 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/2567223221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:34:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:56 smithi136 ceph-mon[25575]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:57.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:56 smithi136 ceph-mon[25575]: mgrmap e32: y(active, since 28s), standbys: x 2024-02-10T07:34:57.231 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:56 smithi064 ceph-mon[24760]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:57.232 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:56 smithi064 ceph-mon[24760]: mgrmap e32: y(active, since 28s), standbys: x 2024-02-10T07:34:57.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:56 smithi064 ceph-mon[29477]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:57.232 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:56 smithi064 ceph-mon[29477]: mgrmap e32: y(active, since 28s), standbys: x 2024-02-10T07:34:57.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:34:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:34:57] "GET /metrics HTTP/1.1" 200 38236 "" "Prometheus/2.33.4" 2024-02-10T07:34:58.732 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: Upgrade: Setting container_image for all mgr 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T07:34:58.733 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: Upgrade: It appears safe to stop mon.a 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:58.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: Upgrade: Setting container_image for all mgr 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T07:34:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:34:58.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T07:34:58.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: Upgrade: It appears safe to stop mon.a 2024-02-10T07:34:58.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:34:58 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:58.978 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:34:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:34:58.978 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: Upgrade: Setting container_image for all mgr 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: Upgrade: It appears safe to stop mon.a 2024-02-10T07:34:58.979 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:34:58 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: Upgrade: Updating mon.a 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: Deploying daemon mon.a on smithi064 2024-02-10T07:35:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[24760]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: Upgrade: Updating mon.a 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: Deploying daemon mon.a on smithi064 2024-02-10T07:35:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:00 smithi064 ceph-mon[29477]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: Upgrade: Updating mon.a 2024-02-10T07:35:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='mgr.y' 2024-02-10T07:35:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:00.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: from='mgr.25189 172.21.15.64:0/4181175491' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: Deploying daemon mon.a on smithi064 2024-02-10T07:35:00.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:00 smithi136 ceph-mon[25575]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:01 smithi064 systemd[1]: Stopping Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a[24756]: 2024-02-10T07:35:01.446+0000 7f8069894700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:35:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:01 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a[24756]: 2024-02-10T07:35:01.446+0000 7f8069894700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T07:35:01.997 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:01 smithi064 podman[64304]: 2024-02-10 07:35:01.702134277 +0000 UTC m=+0.332276108 container died 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_CLEAN=True, release=754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD) 2024-02-10T07:35:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 podman[64304]: 2024-02-10 07:35:02.096398241 +0000 UTC m=+0.726540063 container cleanup 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, version=8, architecture=x86_64, release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-02-10T07:35:02.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 bash[64304]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a 2024-02-10T07:35:02.670 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 podman[64317]: 2024-02-10 07:35:02.339205464 +0000 UTC m=+0.641638536 container remove 0975e381bf03b57a0213025236d8ca4211ed2ba7a4b238066f18fb9afe2b3cd4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_CLEAN=True, version=8, ceph=True, maintainer=Guillaume Abrioux , name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, architecture=x86_64, distribution-scope=public, io.openshift.tags=base centos centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T07:35:02.670 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T07:35:02.670 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 systemd[1]: Stopped Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:35:02.670 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service: Consumed 13.887s CPU time. 2024-02-10T07:35:02.670 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 systemd[1]: Starting Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:03.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 podman[64451]: 2024-02-10 07:35:02.881736944 +0000 UTC m=+0.110810049 container create 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T07:35:03.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:02 smithi064 podman[64451]: 2024-02-10 07:35:02.788808361 +0000 UTC m=+0.017881432 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:35:03.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 podman[64451]: 2024-02-10 07:35:03.150995298 +0000 UTC m=+0.380068361 container init 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:35:03.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 podman[64451]: 2024-02-10 07:35:03.1541192 +0000 UTC m=+0.383192265 container start 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: pidfile_write: ignore empty --pid-file 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: load: jerasure load: lrc 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: RocksDB version: 7.9.2 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Git sha 0 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: DB SUMMARY 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: DB Session ID: IX7ZDMOZ7MIP7E30OLC5 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-02-10T07:35:03.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3514083 ; 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.env: 0x559ed997fae0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.info_log: 0x559edad798c0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.statistics: (nil) 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.use_fsync: 0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:35:03.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.db_log_dir: 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.wal_dir: 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:35:03.317 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.write_buffer_manager: 0x559edad4cdc0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:35:03.318 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.unordered_write: 0 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.row_cache: None 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.wal_filter: None 2024-02-10T07:35:03.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:35:03.320 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:35:03.320 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:35:03.320 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:35:03.320 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.wal_compression: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:35:03.321 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:35:03.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_open_files: -1 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Compression algorithms supported: 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kZSTD supported: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:35:03.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.merge_operator: 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_filter: None 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:35:03.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559edad799c0) 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x559edad7d090 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-02-10T07:35:03.325 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2024-02-10T07:35:03.326 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression: NoCompression 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:35:03.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.num_levels: 7 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:35:03.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:35:03.329 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T07:35:03.330 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.ttl: 2592000 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12091, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cacce143-0300-42d2-8488-c30d2e2fe238 2024-02-10T07:35:03.333 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550503183103, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550503192353, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3219024, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12092, "largest_seqno": 13671, "table_properties": {"data_size": 3211591, "index_size": 4686, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1733, "raw_key_size": 16240, "raw_average_key_size": 23, "raw_value_size": 3196599, "raw_average_value_size": 4646, "num_data_blocks": 212, "num_entries": 688, "num_filter_entries": 688, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707550503, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cacce143-0300-42d2-8488-c30d2e2fe238", "db_session_id": "IX7ZDMOZ7MIP7E30OLC5", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550503192458, "job": 1, "event": "recovery_finished"} 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559edae6e000 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: DB pointer 0x559edae58000 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:03.334 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 1/0 3.07 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 692.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: L6 1/0 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 10.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 692.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 692.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:35:03.335 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 692.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:35:03.336 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 223.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 223.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x559edad7d090#2 capacity: 512.00 MB usage: 6.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???) e3 preinit fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).mds e1 new map 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).mds e1 print_map 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-02-10T07:35:03.337 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 ceph-mon[64470]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 bash[64451]: 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b 2024-02-10T07:35:03.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:03 smithi064 systemd[1]: Started Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (91s) 33s ago 6m 14.2M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (58s) 13s ago 6m 47.6M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 unknown 33s ago 5m - - 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (18s) 13s ago 10m 65.3M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (53s) 33s ago 12m 472M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:35:06.063 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 starting - - - 2048M 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (11m) 13s ago 11m 45.3M 2048M 17.2.0 e1d6a67b021e 6cc41643f620 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (11m) 33s ago 11m 47.6M 2048M 17.2.0 e1d6a67b021e ecae739aa281 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (101s) 33s ago 7m 11.4M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (73s) 13s ago 7m 17.0M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (10m) 33s ago 10m 54.5M 2736M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (10m) 33s ago 10m 55.7M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (9m) 33s ago 9m 50.1M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (9m) 33s ago 9m 54.2M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (9m) 13s ago 9m 52.1M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (8m) 13s ago 8m 48.9M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (8m) 13s ago 8m 50.8M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (7m) 13s ago 7m 53.8M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (23s) 13s ago 7m 44.8M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:35:06.064 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (5m) 33s ago 5m 72.2M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:35:06.065 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (5m) 13s ago 5m 71.9M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:35:06.065 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (5m) 33s ago 5m 70.9M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:35:06.065 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (5m) 13s ago 5m 71.0M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:35:06.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ignoring --setuser ceph since I am not root 2024-02-10T07:35:06.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ignoring --setgroup ceph since I am not root 2024-02-10T07:35:06.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mgr[60184]: -- 172.21.15.64:0/826115467 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56296f36a1a0 con 0x56296f340c00 2024-02-10T07:35:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: mon.a calling monitor election 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: fsmap 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: mgrmap e32: y(active, since 38s), standbys: x 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: from='mgr.25189 ' entity='' 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: mgrmap e33: y(active, since 38s), standbys: x 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-10T07:35:06.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[64470]: daemon iscsi.foo.smithi064.yotmxt on smithi064 is in unknown state 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: mon.a calling monitor election 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: fsmap 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:35:06.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: mgrmap e32: y(active, since 38s), standbys: x 2024-02-10T07:35:06.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: from='mgr.25189 ' entity='' 2024-02-10T07:35:06.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: mgrmap e33: y(active, since 38s), standbys: x 2024-02-10T07:35:06.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-10T07:35:06.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-10T07:35:06.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-mon[29477]: daemon iscsi.foo.smithi064.yotmxt on smithi064 is in unknown state 2024-02-10T07:35:06.479 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": {}, 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:35:06.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:06.481 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:35:06.481 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-02-10T07:35:06.481 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:35:06.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:35:06.481 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:35:06.725 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: ignoring --setuser ceph since I am not root 2024-02-10T07:35:06.725 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: ignoring --setgroup ceph since I am not root 2024-02-10T07:35:06.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: mon.a calling monitor election 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: fsmap 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: mgrmap e32: y(active, since 38s), standbys: x 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: from='mgr.25189 ' entity='' 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: mgrmap e33: y(active, since 38s), standbys: x 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-10T07:35:06.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-10T07:35:06.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-mon[25575]: daemon iscsi.foo.smithi064.yotmxt on smithi064 is in unknown state 2024-02-10T07:35:07.222 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:06.812+0000 7f4e8d5f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:35:07.223 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:06.905+0000 7f4e8d5f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:35:07.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:06.842+0000 7f6e4311b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:35:07.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:06 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:06.940+0000 7f6e4311b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:35:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3332570213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-mon[64470]: mgrmap e34: y(active, since 39s), standbys: x 2024-02-10T07:35:07.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/3332570213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:07.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-mon[29477]: mgrmap e34: y(active, since 39s), standbys: x 2024-02-10T07:35:07.634 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:07.542+0000 7f4e8d5f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:35:07.635 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/3332570213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:07.635 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-mon[25575]: mgrmap e34: y(active, since 39s), standbys: x 2024-02-10T07:35:07.943 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:07.603+0000 7f6e4311b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:35:07.944 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:07.698+0000 7f6e4311b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:35:07.944 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:07.842+0000 7f6e4311b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:35:07.956 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:07.634+0000 7f4e8d5f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:35:07.956 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:07.778+0000 7f4e8d5f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:35:07.956 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:07.873+0000 7f4e8d5f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:35:07.956 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:07.955+0000 7f4e8d5f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:35:08.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:07.942+0000 7f6e4311b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:35:08.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:08.028+0000 7f6e4311b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:35:08.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:08.217+0000 7f4e8d5f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:35:08.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:08.325+0000 7f4e8d5f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:35:08.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:08.411+0000 7f4e8d5f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:35:08.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:08.297+0000 7f6e4311b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:35:08.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:08.413+0000 7f6e4311b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:35:08.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:08.502+0000 7f6e4311b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:35:08.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:08.505+0000 7f4e8d5f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:35:08.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:08.681+0000 7f4e8d5f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:35:09.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:08.599+0000 7f6e4311b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:35:09.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:08.778+0000 7f6e4311b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:35:09.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:09.630+0000 7f4e8d5f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:35:09.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:09.760+0000 7f4e8d5f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:35:10.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:09 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:09.772+0000 7f6e4311b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:35:10.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:09 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:09.908+0000 7f6e4311b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:35:10.378 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:10.092+0000 7f4e8d5f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:35:10.378 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:10.182+0000 7f4e8d5f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:35:10.548 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:10.249+0000 7f6e4311b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:35:10.548 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:10.343+0000 7f6e4311b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:35:10.548 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:10.546+0000 7f6e4311b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:35:10.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:10.377+0000 7f4e8d5f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:35:10.723 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:10.561+0000 7f4e8d5f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:35:10.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:10.739+0000 7f6e4311b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:35:11.106 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:10.847+0000 7f4e8d5f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:35:11.302 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:11.036+0000 7f6e4311b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:35:11.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:11.105+0000 7f4e8d5f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:35:11.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:11.341+0000 7f4e8d5f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:35:11.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:11.301+0000 7f6e4311b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:35:11.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:11 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:11.548+0000 7f6e4311b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:35:12.472 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:12.146+0000 7f4e8d5f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:35:12.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:12.392+0000 7f6e4311b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:35:12.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:12.716+0000 7f4e8d5f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:35:12.972 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:12.816+0000 7f4e8d5f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:35:13.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:12.979+0000 7f6e4311b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:35:13.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:13.084+0000 7f6e4311b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[29477]: Standby manager daemon x restarted 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[29477]: Standby manager daemon x started 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[29477]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[64470]: Standby manager daemon x restarted 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[64470]: Standby manager daemon x started 2024-02-10T07:35:13.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[64470]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:35:13.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[64470]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:35:13.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[64470]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:35:13.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-mon[64470]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:35:13.334 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-mon[25575]: Standby manager daemon x restarted 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-mon[25575]: Standby manager daemon x started 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-mon[25575]: from='mgr.? 172.21.15.136:0/3294138321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:13.035+0000 7f4e8d5f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: [10/Feb/2024:07:35:13] ENGINE Bus STARTING 2024-02-10T07:35:13.335 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: CherryPy Checker: 2024-02-10T07:35:13.336 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: The Application mounted at '' has an empty config. 2024-02-10T07:35:13.336 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: 2024-02-10T07:35:13.336 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: [10/Feb/2024:07:35:13] ENGINE Serving on http://:::9283 2024-02-10T07:35:13.336 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x[54281]: [10/Feb/2024:07:35:13] ENGINE Bus STARTED 2024-02-10T07:35:13.573 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c[29472]: 2024-02-10T07:35:13.335+0000 7f682598f700 -1 mon.c@1(peon).osd e105 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-10T07:35:13.574 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:13.317+0000 7f6e4311b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:35:13.574 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:13] ENGINE Bus STARTING 2024-02-10T07:35:13.574 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: CherryPy Checker: 2024-02-10T07:35:13.587 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b[25564]: 2024-02-10T07:35:13.334+0000 7ffa27e0e700 -1 mon.b@2(peon).osd e105 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-10T07:35:14.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: The Application mounted at '' has an empty config. 2024-02-10T07:35:14.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:14.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:13] ENGINE Serving on http://:::9283 2024-02-10T07:35:14.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:13] ENGINE Bus STARTED 2024-02-10T07:35:14.353 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "", 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:35:14.354 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: mgrmap e35: y(active, since 45s), standbys: x 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: Active manager daemon y restarted 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: Activating manager daemon y 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: mgrmap e36: y(active, starting, since 0.00846648s), standbys: x 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:35:14.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: Manager daemon y is now available 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:14.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:14.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:35:14.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:35:14.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:14.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:14.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:35:14.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:14 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:35:14.548 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:14.346+0000 7f6df9319700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: mgrmap e35: y(active, since 45s), standbys: x 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: Active manager daemon y restarted 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: Activating manager daemon y 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: mgrmap e36: y(active, starting, since 0.00846648s), standbys: x 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:35:14.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: Manager daemon y is now available 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:14.550 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: mgrmap e35: y(active, since 45s), standbys: x 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: Active manager daemon y restarted 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: Activating manager daemon y 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: mgrmap e36: y(active, starting, since 0.00846648s), standbys: x 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:35:14.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:35:14.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: Manager daemon y is now available 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:35:14.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:14 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:35:14.764 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-10T07:35:14.765 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-10T07:35:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: daemon iscsi.foo.smithi064.yotmxt on smithi064 is in unknown state 2024-02-10T07:35:15.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: mgrmap e37: y(active, since 1.01074s), standbys: x 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1397805238' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: mgrmap e37: y(active, since 1.01074s), standbys: x 2024-02-10T07:35:15.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:15 smithi064 ceph-mon[29477]: from='client.? 172.21.15.64:0/1397805238' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: failed to encode map e105 with expected crc 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: mgrmap e37: y(active, since 1.01074s), standbys: x 2024-02-10T07:35:15.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:15 smithi136 ceph-mon[25575]: from='client.? 172.21.15.64:0/1397805238' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:35:16.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[64470]: [10/Feb/2024:07:35:14] ENGINE Bus STARTING 2024-02-10T07:35:16.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[64470]: [10/Feb/2024:07:35:14] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:35:16.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[64470]: [10/Feb/2024:07:35:14] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T07:35:16.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[64470]: [10/Feb/2024:07:35:14] ENGINE Bus STARTED 2024-02-10T07:35:16.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[64470]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-10T07:35:16.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:35:16.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[29477]: [10/Feb/2024:07:35:14] ENGINE Bus STARTING 2024-02-10T07:35:16.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[29477]: [10/Feb/2024:07:35:14] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:35:16.287 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[29477]: [10/Feb/2024:07:35:14] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T07:35:16.288 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[29477]: [10/Feb/2024:07:35:14] ENGINE Bus STARTED 2024-02-10T07:35:16.288 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[29477]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-10T07:35:16.288 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:16 smithi064 ceph-mon[29477]: Cluster is now healthy 2024-02-10T07:35:16.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:16 smithi136 ceph-mon[25575]: [10/Feb/2024:07:35:14] ENGINE Bus STARTING 2024-02-10T07:35:16.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:16 smithi136 ceph-mon[25575]: [10/Feb/2024:07:35:14] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T07:35:16.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:16 smithi136 ceph-mon[25575]: [10/Feb/2024:07:35:14] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T07:35:16.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:16 smithi136 ceph-mon[25575]: [10/Feb/2024:07:35:14] ENGINE Bus STARTED 2024-02-10T07:35:16.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:16 smithi136 ceph-mon[25575]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-10T07:35:16.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:16 smithi136 ceph-mon[25575]: Cluster is now healthy 2024-02-10T07:35:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:17 smithi064 ceph-mon[64470]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:35:17.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:17 smithi064 ceph-mon[64470]: mgrmap e38: y(active, since 3s), standbys: x 2024-02-10T07:35:17.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:17 smithi064 ceph-mon[29477]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:35:17.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:17 smithi064 ceph-mon[29477]: mgrmap e38: y(active, since 3s), standbys: x 2024-02-10T07:35:17.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:17 smithi136 ceph-mon[25575]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:35:17.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:17 smithi136 ceph-mon[25575]: mgrmap e38: y(active, since 3s), standbys: x 2024-02-10T07:35:17.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:35:17] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-02-10T07:35:18.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:35:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:35:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:35:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[64470]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:35:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[64470]: mgrmap e39: y(active, since 4s), standbys: x 2024-02-10T07:35:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[29477]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:35:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[29477]: mgrmap e39: y(active, since 4s), standbys: x 2024-02-10T07:35:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:19 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:19 smithi136 ceph-mon[25575]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:35:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:19 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:19 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:19 smithi136 ceph-mon[25575]: mgrmap e39: y(active, since 4s), standbys: x 2024-02-10T07:35:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:19 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:19 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:20 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:20 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:20 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:20 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:20.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:20 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:21.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:21 smithi136 ceph-mon[25575]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-10T07:35:21.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:21 smithi064 ceph-mon[64470]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-10T07:35:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:21 smithi064 ceph-mon[29477]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-10T07:35:23.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:23 smithi136 ceph-mon[25575]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T07:35:23.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:23 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:23.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:23 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:23.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:23 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:23.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:23 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[64470]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T07:35:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[29477]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T07:35:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:23 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:24.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:35:24.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:35:24.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:24 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.conf 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:35:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:24 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: Upgrade: It appears safe to stop mon.c 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[29477]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:35:25.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:35:25.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:35:25.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: Upgrade: It appears safe to stop mon.c 2024-02-10T07:35:25.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:25.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:25.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:25.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:25.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: Updating smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:35:25.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: Updating smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:35:25.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T07:35:25.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: Upgrade: It appears safe to stop mon.c 2024-02-10T07:35:25.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:25.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:25.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:25.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:25 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:26.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:26 smithi064 ceph-mon[29477]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:26.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:26 smithi064 ceph-mon[29477]: Upgrade: Updating mon.c 2024-02-10T07:35:26.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:26 smithi064 ceph-mon[29477]: Deploying daemon mon.c on smithi064 2024-02-10T07:35:26.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:26 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:26.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:26 smithi064 ceph-mon[64470]: Upgrade: Updating mon.c 2024-02-10T07:35:26.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:26 smithi064 ceph-mon[64470]: Deploying daemon mon.c on smithi064 2024-02-10T07:35:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:26 smithi136 ceph-mon[25575]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:26 smithi136 ceph-mon[25575]: Upgrade: Updating mon.c 2024-02-10T07:35:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:26 smithi136 ceph-mon[25575]: Deploying daemon mon.c on smithi064 2024-02-10T07:35:27.196 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:26 smithi064 systemd[1]: Stopping Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:27.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:35:27] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-02-10T07:35:27.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c[29472]: 2024-02-10T07:35:27.195+0000 7f682b19a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:35:27.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c[29472]: 2024-02-10T07:35:27.195+0000 7f682b19a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T07:35:27.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:27 smithi064 podman[66850]: 2024-02-10 07:35:27.338535168 +0000 UTC m=+0.253943776 container died ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, io.buildah.version=1.19.8, vcs-type=git, io.openshift.tags=base centos centos-stream, release=754, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-02-10T07:35:27.988 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:27 smithi064 podman[66850]: 2024-02-10 07:35:27.695346449 +0000 UTC m=+0.610755061 container cleanup ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, release=754, ceph=True, vcs-type=git, distribution-scope=public, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-10T07:35:27.988 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:27 smithi064 bash[66850]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c 2024-02-10T07:35:28.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:27 smithi064 podman[66862]: 2024-02-10 07:35:27.988208976 +0000 UTC m=+0.645112910 container remove ecae739aa2812072db564c42b69494dba2d69de0c7cb800d89759d680c488a0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, architecture=x86_64, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, RELEASE=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, version=8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, distribution-scope=public) 2024-02-10T07:35:28.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-10T07:35:28.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 systemd[1]: Stopped Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:35:28.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c.service: Consumed 8.098s CPU time. 2024-02-10T07:35:28.691 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 systemd[1]: Starting Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:28.691 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 podman[66998]: 2024-02-10 07:35:28.44564871 +0000 UTC m=+0.017721022 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:35:28.691 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 podman[66998]: 2024-02-10 07:35:28.564336916 +0000 UTC m=+0.136409212 container create cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T07:35:28.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:35:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:35:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:35:29.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 podman[66998]: 2024-02-10 07:35:28.800574449 +0000 UTC m=+0.372646746 container init cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T07:35:29.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 podman[66998]: 2024-02-10 07:35:28.803961299 +0000 UTC m=+0.376033596 container start cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-10T07:35:29.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:35:29.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T07:35:29.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: pidfile_write: ignore empty --pid-file 2024-02-10T07:35:29.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: load: jerasure load: lrc 2024-02-10T07:35:29.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: RocksDB version: 7.9.2 2024-02-10T07:35:29.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Git sha 0 2024-02-10T07:35:29.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T07:35:29.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: DB SUMMARY 2024-02-10T07:35:29.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: DB Session ID: H4PZVCMGY2FHKJ4YWAZP 2024-02-10T07:35:29.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:35:29.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:35:29.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-02-10T07:35:29.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-02-10T07:35:29.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 279260 ; 2024-02-10T07:35:29.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:35:29.068 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.env: 0x55f49c6c5ae0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.info_log: 0x55f49d0bb8c0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.statistics: (nil) 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.use_fsync: 0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:35:29.069 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.db_log_dir: 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.wal_dir: 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:35:29.070 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.write_buffer_manager: 0x55f49d08edc0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.unordered_write: 0 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:35:29.071 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.row_cache: None 2024-02-10T07:35:29.072 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.wal_filter: None 2024-02-10T07:35:29.072 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.wal_compression: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:35:29.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_open_files: -1 2024-02-10T07:35:29.075 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Compression algorithms supported: 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kZSTD supported: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:35:29.076 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.merge_operator: 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_filter: None 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f49d0bb9c0) 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:35:29.077 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_type: 0 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_index_type: 0 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_shortening: 1 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: checksum: 4 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: no_block_cache: 0 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache: 0x55f49d0bf090 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_options: 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: capacity : 536870912 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_shard_bits : 4 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: strict_capacity_limit : 0 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_compressed: (nil) 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: persistent_cache: (nil) 2024-02-10T07:35:29.078 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size: 4096 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size_deviation: 10 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_restart_interval: 16 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_block_restart_interval: 1 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: metadata_block_size: 4096 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: partition_filters: 0 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: use_delta_encoding: 1 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: filter_policy: bloomfilter 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: whole_key_filtering: 1 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: verify_compression: 0 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: format_version: 5 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_index_compression: 1 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_align: 0 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: max_auto_readahead_size: 262144 2024-02-10T07:35:29.079 INFO:journalctl@ceph.mon.c.smithi064.stdout: prepopulate_block_cache: 0 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout: initial_auto_readahead_size: 8192 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression: NoCompression 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.num_levels: 7 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:35:29.080 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:35:29.081 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:35:29.082 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:35:29.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:35:29.084 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.ttl: 2592000 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:35:29.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15185, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8ed8990b-d744-449f-96ba-44fc8f21e071 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550528832384, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550528833791, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 186433, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15190, "largest_seqno": 15300, "table_properties": {"data_size": 184781, "index_size": 448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1759, "raw_average_key_size": 27, "raw_value_size": 183153, "raw_average_value_size": 2817, "num_data_blocks": 18, "num_entries": 65, "num_filter_entries": 65, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707550528, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8ed8990b-d744-449f-96ba-44fc8f21e071", "db_session_id": "H4PZVCMGY2FHKJ4YWAZP", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550528833875, "job": 1, "event": "recovery_finished"} 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-10T07:35:29.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f49d1b0000 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: DB pointer 0x55f49d19a000 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** DB Stats ** 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:35:29.087 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: L0 1/0 182.06 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 193.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: L6 1/0 11.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: Sum 2/0 11.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 193.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 193.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 193.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:35:29.088 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative compaction: 0.00 GB write, 52.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval compaction: 0.00 GB write, 52.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache BinnedLRUCache@0x55f49d0bf090#2 capacity: 512.00 MB usage: 0.75 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.52 KB,9.83477e-05%) Misc(1,0.00 KB,0%) 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: starting mon.c rank 1 at public addrs [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] at bind addrs [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???) e3 preinit fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:35:29.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).mds e1 new map 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).mds e1 print_map 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout: e1 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout: legacy client fscid: -1 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout: No filesystems configured 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 ceph-mon[67016]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 bash[66998]: cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 2024-02-10T07:35:29.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:28 smithi064 systemd[1]: Started Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:35:30.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:30 smithi064 ceph-mon[64470]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T07:35:30.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:30 smithi064 ceph-mon[67016]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T07:35:30.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:30 smithi136 ceph-mon[25575]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T07:35:31.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: mon.c calling monitor election 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: mon.a calling monitor election 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: fsmap 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: mgrmap e39: y(active, since 16s), standbys: x 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: overall HEALTH_OK 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: mon.c calling monitor election 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: mon.a calling monitor election 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: fsmap 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: mgrmap e39: y(active, since 16s), standbys: x 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: overall HEALTH_OK 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:31.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.317 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:31 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: mon.c calling monitor election 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: mon.a calling monitor election 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: fsmap 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: mgrmap e39: y(active, since 16s), standbys: x 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: overall HEALTH_OK 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:31.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:31 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:33 smithi064 ceph-mon[64470]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:33.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:33 smithi064 ceph-mon[67016]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:33 smithi136 ceph-mon[25575]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:34.176 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:33.841+0000 7f6df9319700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-10T07:35:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:35 smithi064 ceph-mon[64470]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:35 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:35 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:35 smithi064 ceph-mon[67016]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:35 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:35 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:35 smithi136 ceph-mon[25575]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:35 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:35 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:36.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:36 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:36.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:36 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:36.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:36 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:36.538 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:36 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:36.539 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:36 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:36.539 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:36 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:36.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:36 smithi136 ceph-mon[25575]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:36.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:36 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:36.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:36 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:37.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:37 smithi064 ceph-mon[64470]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:37.538 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:37 smithi064 ceph-mon[67016]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:37.539 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:35:37] "GET /metrics HTTP/1.1" 200 38189 "" "Prometheus/2.33.4" 2024-02-10T07:35:37.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:37 smithi136 ceph-mon[25575]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:38.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:35:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:35:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:35:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[64470]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[67016]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:39 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:39 smithi136 ceph-mon[25575]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:39.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:39 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:39 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:39.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:39 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:39.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:39 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:39.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:39 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:40 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:40 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:40 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:35:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:40 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T07:35:40.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:40.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:40.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:35:40.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T07:35:40.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:35:40.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:35:40.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T07:35:40.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-10T07:35:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[64470]: Upgrade: It appears safe to stop mon.b 2024-02-10T07:35:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[64470]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[64470]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[67016]: Upgrade: It appears safe to stop mon.b 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[67016]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[67016]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:41 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:41.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:41 smithi136 ceph-mon[25575]: Upgrade: It appears safe to stop mon.b 2024-02-10T07:35:41.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:41 smithi136 ceph-mon[25575]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:41.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:41 smithi136 ceph-mon[25575]: from='mgr.44106 ' entity='mgr.y' 2024-02-10T07:35:41.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:41 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:41.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:41 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:41.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:41 smithi136 ceph-mon[25575]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:42 smithi136 ceph-mon[25575]: Upgrade: Updating mon.b 2024-02-10T07:35:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:42 smithi136 ceph-mon[25575]: Deploying daemon mon.b on smithi136 2024-02-10T07:35:42.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:42 smithi064 ceph-mon[64470]: Upgrade: Updating mon.b 2024-02-10T07:35:42.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:42 smithi064 ceph-mon[64470]: Deploying daemon mon.b on smithi136 2024-02-10T07:35:42.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:42 smithi064 ceph-mon[67016]: Upgrade: Updating mon.b 2024-02-10T07:35:42.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:42 smithi064 ceph-mon[67016]: Deploying daemon mon.b on smithi136 2024-02-10T07:35:42.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:42 smithi136 systemd[1]: Stopping Ceph mon.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b[25564]: 2024-02-10T07:35:42.985+0000 7ffa2d619700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:35:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b[25564]: 2024-02-10T07:35:42.985+0000 7ffa2d619700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T07:35:43.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:42] ENGINE Bus STOPPING 2024-02-10T07:35:43.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:35:43.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:43] ENGINE Bus STOPPED 2024-02-10T07:35:43.315 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:43] ENGINE Bus STARTING 2024-02-10T07:35:43.315 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:43] ENGINE Serving on http://:::9283 2024-02-10T07:35:43.315 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:35:43] ENGINE Bus STARTED 2024-02-10T07:35:43.537 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:43 smithi136 podman[57722]: 2024-02-10 07:35:43.24083847 +0000 UTC m=+0.378349114 container died 6cc41643f6204425093dd3680e140016d43d12678e31960b6dbe4d34053ca46b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, ceph=True, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T07:35:43.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:43 smithi136 podman[57722]: 2024-02-10 07:35:43.644817414 +0000 UTC m=+0.782328064 container cleanup 6cc41643f6204425093dd3680e140016d43d12678e31960b6dbe4d34053ca46b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, vcs-type=git, com.redhat.component=centos-stream-container, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-02-10T07:35:43.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:43 smithi136 bash[57722]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b 2024-02-10T07:35:44.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:43 smithi136 podman[57736]: 2024-02-10 07:35:43.898238408 +0000 UTC m=+0.655538960 container remove 6cc41643f6204425093dd3680e140016d43d12678e31960b6dbe4d34053ca46b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, version=8, ceph=True, architecture=x86_64, RELEASE=HEAD, name=centos-stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux ) 2024-02-10T07:35:44.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.b.service: Deactivated successfully. 2024-02-10T07:35:44.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 systemd[1]: Stopped Ceph mon.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:35:44.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.b.service: Consumed 9.277s CPU time. 2024-02-10T07:35:44.563 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 systemd[1]: Starting Ceph mon.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:44.563 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 podman[57869]: 2024-02-10 07:35:44.430483204 +0000 UTC m=+0.019110827 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:35:44.835 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 podman[57869]: 2024-02-10 07:35:44.564051944 +0000 UTC m=+0.152679560 container create c583d22115e375a89fe3c804e87a4e83bd8642fef39ec11ab9ad1a185be5a71c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-10T07:35:44.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 podman[57869]: 2024-02-10 07:35:44.808540369 +0000 UTC m=+0.397167981 container init c583d22115e375a89fe3c804e87a4e83bd8642fef39ec11ab9ad1a185be5a71c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-10T07:35:44.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 podman[57869]: 2024-02-10 07:35:44.811595975 +0000 UTC m=+0.400223591 container start c583d22115e375a89fe3c804e87a4e83bd8642fef39ec11ab9ad1a185be5a71c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-10T07:35:45.136 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:35:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:35:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T07:35:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: pidfile_write: ignore empty --pid-file 2024-02-10T07:35:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: load: jerasure load: lrc 2024-02-10T07:35:45.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: RocksDB version: 7.9.2 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Git sha 0 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: DB SUMMARY 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: DB Session ID: QF204IXXO7EVEND3KJD0 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 974715 ; 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:35:45.224 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.env: 0x55a313ff5ae0 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.info_log: 0x55a3161738c0 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.statistics: (nil) 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.use_fsync: 0 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:35:45.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.db_log_dir: 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.wal_dir: 2024-02-10T07:35:45.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:35:45.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.write_buffer_manager: 0x55a316146dc0 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.unordered_write: 0 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:35:45.229 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.row_cache: None 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.wal_filter: None 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.wal_compression: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:35:45.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:35:45.231 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_open_files: -1 2024-02-10T07:35:45.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Compression algorithms supported: 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kZSTD supported: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:35:45.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.merge_operator: 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_filter: None 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:35:45.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a3161739c0) 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_type: 0 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: data_block_index_type: 0 2024-02-10T07:35:45.235 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_shortening: 1 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: checksum: 4 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: no_block_cache: 0 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache: 0x55a316177090 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_options: 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: capacity : 536870912 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: num_shard_bits : 4 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: strict_capacity_limit : 0 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_compressed: (nil) 2024-02-10T07:35:45.236 INFO:journalctl@ceph.mon.b.smithi136.stdout: persistent_cache: (nil) 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_size: 4096 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_size_deviation: 10 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_restart_interval: 16 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_block_restart_interval: 1 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: metadata_block_size: 4096 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: partition_filters: 0 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: use_delta_encoding: 1 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: filter_policy: bloomfilter 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: whole_key_filtering: 1 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: verify_compression: 0 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: format_version: 5 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_index_compression: 1 2024-02-10T07:35:45.237 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_align: 0 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout: max_auto_readahead_size: 262144 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout: prepopulate_block_cache: 0 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout: initial_auto_readahead_size: 8192 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression: NoCompression 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:35:45.238 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.num_levels: 7 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:35:45.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:35:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:35:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:35:45.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:35:45.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:35:45.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:35:45.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:35:45.245 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:35:45.246 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.ttl: 2592000 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T07:35:45.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15095, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ac233f77-2f6a-46be-8568-b0e141509236 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550544840987, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550544844197, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 586736, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15100, "largest_seqno": 15777, "table_properties": {"data_size": 583454, "index_size": 1179, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1093, "raw_key_size": 8743, "raw_average_key_size": 20, "raw_value_size": 576070, "raw_average_value_size": 1324, "num_data_blocks": 53, "num_entries": 435, "num_filter_entries": 435, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707550544, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ac233f77-2f6a-46be-8568-b0e141509236", "db_session_id": "QF204IXXO7EVEND3KJD0", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550544844282, "job": 1, "event": "recovery_finished"} 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-10T07:35:45.248 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a316268000 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: DB pointer 0x55a316252000 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** DB Stats ** 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:35:45.249 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: L0 1/0 572.98 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 336.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: L6 1/0 11.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: Sum 2/0 11.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 336.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 336.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 336.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T07:35:45.250 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative compaction: 0.00 GB write, 105.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval compaction: 0.00 GB write, 105.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Block cache BinnedLRUCache@0x55a316177090#2 capacity: 512.00 MB usage: 2.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,1.33 KB,0.00025332%) Misc(1,0.00 KB,0%) 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: starting mon.b rank 2 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???) e3 preinit fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:35:45.251 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).mds e1 new map 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).mds e1 print_map 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout: e1 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout: legacy client fscid: -1 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout: No filesystems configured 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 ceph-mon[57888]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-02-10T07:35:45.252 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 bash[57869]: c583d22115e375a89fe3c804e87a4e83bd8642fef39ec11ab9ad1a185be5a71c 2024-02-10T07:35:45.253 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:44 smithi136 systemd[1]: Started Ceph mon.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (2m) 11s ago 7m 17.5M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (98s) 27s ago 6m 47.9M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (67s) 11s ago 6m 51.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (58s) 27s ago 11m 446M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (93s) 11s ago 13m 491M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (43s) 11s ago 13m 39.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 starting - - - 2048M 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (17s) 11s ago 12m 17.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (2m) 11s ago 7m 19.3M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (113s) 27s ago 7m 19.1M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:35:45.803 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (11m) 11s ago 11m 55.6M 2736M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (10m) 11s ago 10m 56.6M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (10m) 11s ago 10m 50.9M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (10m) 11s ago 10m 55.2M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (9m) 27s ago 9m 52.7M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (9m) 27s ago 9m 49.4M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (8m) 27s ago 8m 51.0M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (8m) 27s ago 8m 54.2M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (63s) 27s ago 7m 53.7M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (6m) 11s ago 6m 73.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (6m) 27s ago 6m 72.2M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (6m) 11s ago 6m 72.2M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:35:45.804 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (6m) 27s ago 6m 71.5M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:35:46.211 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:35:46.212 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:35:46.212 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T07:35:46.212 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:35:46.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:46.212 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:35:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:35:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:35:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:35:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:35:46.214 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:35:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:35:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "4/23 daemons upgraded", 2024-02-10T07:35:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T07:35:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:35:46.561 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:35:46.963 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:35:47.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:47.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='client.54109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:47.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:47.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: overall HEALTH_OK 2024-02-10T07:35:47.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:35:47.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: mon.a calling monitor election 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: mon.c calling monitor election 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: mon.b calling monitor election 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: monmap e4: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: fsmap 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: mgrmap e39: y(active, since 32s), standbys: x 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: overall HEALTH_OK 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:47.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:47 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/3625368526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:35:47.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:35:47] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.33.4" 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='client.54109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: overall HEALTH_OK 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: mon.a calling monitor election 2024-02-10T07:35:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: mon.c calling monitor election 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: mon.b calling monitor election 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: monmap e4: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: fsmap 2024-02-10T07:35:47.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:47.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: mgrmap e39: y(active, since 32s), standbys: x 2024-02-10T07:35:47.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: overall HEALTH_OK 2024-02-10T07:35:47.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:47.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:47.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3625368526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:35:47.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:47.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='client.54109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:47.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:47.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: overall HEALTH_OK 2024-02-10T07:35:47.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: mon.a calling monitor election 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: mon.c calling monitor election 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: mon.b calling monitor election 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: monmap e4: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:35:47.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: fsmap 2024-02-10T07:35:47.570 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:35:47.570 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: mgrmap e39: y(active, since 32s), standbys: x 2024-02-10T07:35:47.570 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: overall HEALTH_OK 2024-02-10T07:35:47.570 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:47.570 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:47.570 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:47 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/3625368526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:35:48.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:48 smithi064 ceph-mon[64470]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:48.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:48 smithi064 ceph-mon[67016]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:48.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:48 smithi136 ceph-mon[57888]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:35:48.946 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:35:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:35:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:35:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:49 smithi064 ceph-mon[64470]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:49 smithi064 ceph-mon[67016]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:49.577 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:49 smithi136 ceph-mon[57888]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:50.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:49 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:35:49.851+0000 7f6df9319700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-10T07:35:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:51 smithi064 ceph-mon[64470]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:51 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:51 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:51.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:51 smithi064 ceph-mon[67016]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:51.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:51 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:51.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:51 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:51 smithi136 ceph-mon[57888]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:51 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:51 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:53.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:52 smithi136 ceph-mon[57888]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:52 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:53.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:52 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:52 smithi064 ceph-mon[64470]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:52 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:53.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:52 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:52 smithi064 ceph-mon[67016]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:52 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:53.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:52 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:55.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:55 smithi136 ceph-mon[57888]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:55.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:55 smithi064 ceph-mon[64470]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:55 smithi064 ceph-mon[67016]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:56.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:56.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:56.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:56.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:56 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:56.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:56 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:57.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:35:57] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.33.4" 2024-02-10T07:35:57.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: Reconfiguring mon.a (monmap changed)... 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: Reconfiguring daemon mon.a on smithi064 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:35:57.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: Reconfiguring mon.a (monmap changed)... 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: Reconfiguring daemon mon.a on smithi064 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:35:57.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:57.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: Reconfiguring mon.a (monmap changed)... 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: Reconfiguring daemon mon.a on smithi064 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:35:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:58.500 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:58 smithi064 ceph-mon[64470]: Reconfiguring mgr.y (monmap changed)... 2024-02-10T07:35:58.500 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:58 smithi064 ceph-mon[64470]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T07:35:58.501 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:58 smithi064 ceph-mon[67016]: Reconfiguring mgr.y (monmap changed)... 2024-02-10T07:35:58.501 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:58 smithi064 ceph-mon[67016]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T07:35:58.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:58 smithi136 ceph-mon[57888]: Reconfiguring mgr.y (monmap changed)... 2024-02-10T07:35:58.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:58 smithi136 ceph-mon[57888]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T07:35:58.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:35:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:35:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:59.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:59.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:35:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:35:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:35:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:35:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:35:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:35:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[64470]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:36:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[64470]: Reconfiguring daemon mon.c on smithi064 2024-02-10T07:36:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[67016]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[67016]: Reconfiguring daemon mon.c on smithi064 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:36:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:00.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:00 smithi136 ceph-mon[57888]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:36:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:00 smithi136 ceph-mon[57888]: Reconfiguring daemon mon.c on smithi064 2024-02-10T07:36:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:36:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:01.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:01 smithi064 ceph-mon[64470]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:01.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:01 smithi064 ceph-mon[64470]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T07:36:01.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:01 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.0 on smithi064 2024-02-10T07:36:01.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:01 smithi064 ceph-mon[67016]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:01.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:01 smithi064 ceph-mon[67016]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T07:36:01.471 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:01 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.0 on smithi064 2024-02-10T07:36:01.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:01 smithi136 ceph-mon[57888]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:01.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:01 smithi136 ceph-mon[57888]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T07:36:01.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:01 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.0 on smithi064 2024-02-10T07:36:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:03.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.1 on smithi064 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:03.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:03.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:03.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T07:36:03.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:36:03.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:03.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:02 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.1 on smithi064 2024-02-10T07:36:03.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T07:36:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:36:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:03.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:02 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.1 on smithi064 2024-02-10T07:36:04.733 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:04.733 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.2 on smithi064 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:36:04.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:04.735 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:04 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.2 on smithi064 2024-02-10T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T07:36:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:36:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:04 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.2 on smithi064 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:06.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.3 on smithi064 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:06.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:06 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.3 on smithi064 2024-02-10T07:36:06.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:06.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T07:36:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:36:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:06.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:06 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.3 on smithi064 2024-02-10T07:36:07.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[67016]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:07.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:07.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:07.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:07.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:07.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[64470]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:07.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:07.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:07.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:07.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:07.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:07 smithi136 ceph-mon[57888]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:07.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:07 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:07.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:07 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:07.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:07 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:07.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:07 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:07.675 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:36:07] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-02-10T07:36:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:08 smithi064 ceph-mon[64470]: Reconfiguring rgw.foo.smithi064.elxhce (monmap changed)... 2024-02-10T07:36:08.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:08 smithi064 ceph-mon[64470]: Reconfiguring daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:36:08.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:08 smithi064 ceph-mon[67016]: Reconfiguring rgw.foo.smithi064.elxhce (monmap changed)... 2024-02-10T07:36:08.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:08 smithi064 ceph-mon[67016]: Reconfiguring daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:36:08.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:08 smithi136 ceph-mon[57888]: Reconfiguring rgw.foo.smithi064.elxhce (monmap changed)... 2024-02-10T07:36:08.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:08 smithi136 ceph-mon[57888]: Reconfiguring daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:36:08.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:36:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:36:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[64470]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[67016]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:09.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:09.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:09.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:09.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:09 smithi136 ceph-mon[57888]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:09.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:09.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:09.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:09.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:10.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: Reconfiguring rgw.smpl.smithi064.dqyqnr (monmap changed)... 2024-02-10T07:36:10.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: Reconfiguring daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:36:10.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:10.552 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:10.552 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:36:10.552 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:36:10.552 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: Reconfiguring rgw.smpl.smithi064.dqyqnr (monmap changed)... 2024-02-10T07:36:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: Reconfiguring daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:36:10.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: Reconfiguring rgw.smpl.smithi064.dqyqnr (monmap changed)... 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: Reconfiguring daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:36:10.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:11.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:11 smithi064 ceph-mon[64470]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:11.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:11 smithi064 ceph-mon[64470]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:36:11.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:11 smithi064 ceph-mon[64470]: Reconfiguring daemon mon.b on smithi136 2024-02-10T07:36:11.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:11 smithi064 ceph-mon[67016]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:11.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:11 smithi064 ceph-mon[67016]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:36:11.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:11 smithi064 ceph-mon[67016]: Reconfiguring daemon mon.b on smithi136 2024-02-10T07:36:11.584 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:11 smithi136 ceph-mon[57888]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:11.584 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:11 smithi136 ceph-mon[57888]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:36:11.585 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:11 smithi136 ceph-mon[57888]: Reconfiguring daemon mon.b on smithi136 2024-02-10T07:36:12.961 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:12.961 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:12.961 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:12.961 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: Reconfiguring mgr.x (monmap changed)... 2024-02-10T07:36:12.961 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:36:12.961 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:36:12.962 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:12.962 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:12 smithi136 ceph-mon[57888]: Reconfiguring daemon mgr.x on smithi136 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: Reconfiguring mgr.x (monmap changed)... 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:36:13.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[64470]: Reconfiguring daemon mgr.x on smithi136 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: Reconfiguring mgr.x (monmap changed)... 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:13.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:12 smithi064 ceph-mon[67016]: Reconfiguring daemon mgr.x on smithi136 2024-02-10T07:36:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T07:36:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.4 on smithi136 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.4 on smithi136 2024-02-10T07:36:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T07:36:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:36:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.4 on smithi136 2024-02-10T07:36:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:15.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:15 smithi136 ceph-mon[57888]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:15.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:15.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:15.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:36:15.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[64470]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[67016]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:15.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:36:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:16 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:16 smithi064 ceph-mon[64470]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T07:36:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:16 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.5 on smithi136 2024-02-10T07:36:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:16 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:16 smithi064 ceph-mon[67016]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T07:36:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:16 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.5 on smithi136 2024-02-10T07:36:16.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:16 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:16.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:16 smithi136 ceph-mon[57888]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T07:36:16.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:16 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.5 on smithi136 2024-02-10T07:36:17.324 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:36:17.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:17 smithi136 ceph-mon[57888]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:17.520 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:17.520 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:17.520 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:36:17.520 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[64470]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:36:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:17.566 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:36:17] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-02-10T07:36:17.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[67016]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:17.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:17.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:17.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:36:17.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:17.991 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:36:17.991 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (2m) 43s ago 7m 17.5M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:36:17.991 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (2m) 27s ago 7m 47.3M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:36:17.991 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (99s) 43s ago 6m 51.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (90s) 27s ago 12m 448M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (2m) 43s ago 13m 491M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (75s) 43s ago 13m 39.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (33s) 27s ago 12m 20.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (49s) 43s ago 12m 17.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (2m) 43s ago 8m 19.3M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (2m) 27s ago 8m 17.3M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (11m) 43s ago 11m 55.6M 2736M 17.2.0 e1d6a67b021e e175bda257b1 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (11m) 43s ago 11m 56.6M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (11m) 43s ago 11m 50.9M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (10m) 43s ago 10m 55.2M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (10m) 27s ago 10m 53.1M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (9m) 27s ago 9m 49.9M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:36:17.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (9m) 27s ago 9m 50.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (9m) 27s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (95s) 27s ago 8m 54.8M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (6m) 43s ago 6m 73.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (6m) 27s ago 6m 72.4M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (6m) 43s ago 6m 72.2M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (6m) 27s ago 6m 71.9M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:18.402 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:36:18.403 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:36:18.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:18 smithi064 ceph-mon[64470]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T07:36:18.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:18 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.6 on smithi136 2024-02-10T07:36:18.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:18 smithi064 ceph-mon[64470]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:18.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:18 smithi064 ceph-mon[67016]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T07:36:18.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:18 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.6 on smithi136 2024-02-10T07:36:18.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:18 smithi064 ceph-mon[67016]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:18.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:18 smithi136 ceph-mon[57888]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T07:36:18.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:18 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.6 on smithi136 2024-02-10T07:36:18.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:18 smithi136 ceph-mon[57888]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:18.741 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:36:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:36:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "5/23 daemons upgraded", 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:36:18.932 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:36:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:36:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:36:19.143 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:36:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/306318186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2969941021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/306318186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:19 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2969941021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:36:19.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:19.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/306318186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:19.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:19 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2969941021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:36:20.610 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T07:36:20.610 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: Reconfiguring daemon osd.7 on smithi136 2024-02-10T07:36:20.610 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:20.610 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:20.611 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:20.611 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:20.611 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:20.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T07:36:20.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: Reconfiguring daemon osd.7 on smithi136 2024-02-10T07:36:20.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:20.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: Reconfiguring daemon osd.7 on smithi136 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:20.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:21.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:21 smithi136 ceph-mon[57888]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:21.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:21 smithi136 ceph-mon[57888]: Reconfiguring rgw.foo.smithi136.xkhtnu (monmap changed)... 2024-02-10T07:36:21.620 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:21 smithi136 ceph-mon[57888]: Reconfiguring daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:36:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:21 smithi064 ceph-mon[64470]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:21 smithi064 ceph-mon[64470]: Reconfiguring rgw.foo.smithi136.xkhtnu (monmap changed)... 2024-02-10T07:36:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:21 smithi064 ceph-mon[64470]: Reconfiguring daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:36:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:21 smithi064 ceph-mon[67016]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:21 smithi064 ceph-mon[67016]: Reconfiguring rgw.foo.smithi136.xkhtnu (monmap changed)... 2024-02-10T07:36:21.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:21 smithi064 ceph-mon[67016]: Reconfiguring daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: Reconfiguring rgw.smpl.smithi136.fharky (monmap changed)... 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:22.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:22 smithi136 ceph-mon[57888]: Reconfiguring daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:36:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: Reconfiguring rgw.smpl.smithi136.fharky (monmap changed)... 2024-02-10T07:36:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:23.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[64470]: Reconfiguring daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: Reconfiguring rgw.smpl.smithi136.fharky (monmap changed)... 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:23.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:22 smithi064 ceph-mon[67016]: Reconfiguring daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:36:24.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all mon 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T07:36:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all crash 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[64470]: Upgrade: osd.0 is safe to restart 2024-02-10T07:36:24.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all mon 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T07:36:24.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all crash 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T07:36:24.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:24 smithi064 ceph-mon[67016]: Upgrade: osd.0 is safe to restart 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all mon 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-10T07:36:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all crash 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T07:36:24.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:24 smithi136 ceph-mon[57888]: Upgrade: osd.0 is safe to restart 2024-02-10T07:36:26.138 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[64470]: Upgrade: Updating osd.0 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[64470]: Deploying daemon osd.0 on smithi064 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[67016]: Upgrade: Updating osd.0 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:26.139 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:25 smithi064 ceph-mon[67016]: Deploying daemon osd.0 on smithi064 2024-02-10T07:36:26.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:25 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:26.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:25 smithi136 ceph-mon[57888]: Upgrade: Updating osd.0 2024-02-10T07:36:26.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:25 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:26.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:25 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:36:26.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:25 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:26.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:25 smithi136 ceph-mon[57888]: Deploying daemon osd.0 on smithi064 2024-02-10T07:36:27.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:26 smithi064 ceph-mon[64470]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:27.161 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:26 smithi064 ceph-mon[67016]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:27.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:26 smithi136 ceph-mon[57888]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:27.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:36:27] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-02-10T07:36:28.564 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:28 smithi064 systemd[1]: Stopping Ceph osd.0 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:36:28.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:36:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:36:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:36:29.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[32329]: 2024-02-10T07:36:28.599+0000 7ff67a082700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:36:29.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[32329]: 2024-02-10T07:36:28.599+0000 7ff67a082700 -1 osd.0 105 *** Got signal Terminated *** 2024-02-10T07:36:29.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[32329]: 2024-02-10T07:36:28.599+0000 7ff67a082700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:36:29.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:29 smithi064 ceph-mon[64470]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:29.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:29.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:29 smithi064 ceph-mon[64470]: osd.0 marked itself down and dead 2024-02-10T07:36:29.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:29 smithi064 ceph-mon[67016]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:29.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:29.549 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:29 smithi064 ceph-mon[67016]: osd.0 marked itself down and dead 2024-02-10T07:36:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:29 smithi136 ceph-mon[57888]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:29 smithi136 ceph-mon[57888]: osd.0 marked itself down and dead 2024-02-10T07:36:29.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:29 smithi064 podman[69941]: 2024-02-10 07:36:29.548787001 +0000 UTC m=+1.194449200 container died e175bda257b128fbcb452931f554613ede53b20854a5f511a0f5b1097a57cdf2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.expose-services=, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , release=754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T07:36:30.158 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:29 smithi064 podman[69941]: 2024-02-10 07:36:29.899353132 +0000 UTC m=+1.545015330 container cleanup e175bda257b128fbcb452931f554613ede53b20854a5f511a0f5b1097a57cdf2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , distribution-scope=public, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870) 2024-02-10T07:36:30.159 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:29 smithi064 bash[69941]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0 2024-02-10T07:36:30.532 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:30 smithi064 podman[69959]: 2024-02-10 07:36:30.158698273 +0000 UTC m=+0.609412575 container remove e175bda257b128fbcb452931f554613ede53b20854a5f511a0f5b1097a57cdf2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.buildah.version=1.19.8, version=8, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.component=centos-stream-container) 2024-02-10T07:36:30.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:36:30.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: osdmap e106: 8 total, 7 up, 8 in 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[64470]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: osdmap e106: 8 total, 7 up, 8 in 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:30 smithi064 ceph-mon[67016]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:36:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: osdmap e106: 8 total, 7 up, 8 in 2024-02-10T07:36:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:30 smithi136 ceph-mon[57888]: failed to encode map e106 with expected crc 2024-02-10T07:36:30.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:30 smithi064 podman[70050]: 2024-02-10 07:36:30.435089067 +0000 UTC m=+0.017702996 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:36:30.815 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:30 smithi064 podman[70050]: 2024-02-10 07:36:30.55128198 +0000 UTC m=+0.133895902 container create 4326fee4be41c7b74533a7109bc1632ae2caa1b1f6533a1e23a613a18c9c67e5 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T07:36:31.118 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:30 smithi064 podman[70050]: 2024-02-10 07:36:30.820865392 +0000 UTC m=+0.403479323 container init 4326fee4be41c7b74533a7109bc1632ae2caa1b1f6533a1e23a613a18c9c67e5 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T07:36:31.118 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:30 smithi064 podman[70050]: 2024-02-10 07:36:30.824376252 +0000 UTC m=+0.406990183 container start 4326fee4be41c7b74533a7109bc1632ae2caa1b1f6533a1e23a613a18c9c67e5 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T07:36:31.118 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:30 smithi064 podman[70050]: 2024-02-10 07:36:30.903955053 +0000 UTC m=+0.486568987 container attach 4326fee4be41c7b74533a7109bc1632ae2caa1b1f6533a1e23a613a18c9c67e5 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True) 2024-02-10T07:36:31.440 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: pgmap v42: 161 pgs: 52 peering, 109 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: Health check failed: Reduced data availability: 8 pgs inactive, 50 pgs peering (PG_AVAILABILITY) 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: osdmap e107: 8 total, 7 up, 8 in 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.441 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[64470]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: pgmap v42: 161 pgs: 52 peering, 109 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: Health check failed: Reduced data availability: 8 pgs inactive, 50 pgs peering (PG_AVAILABILITY) 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: osdmap e107: 8 total, 7 up, 8 in 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:31 smithi064 ceph-mon[67016]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.442 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:31 smithi064 podman[70050]: 2024-02-10 07:36:31.118353686 +0000 UTC m=+0.700967620 container died 4326fee4be41c7b74533a7109bc1632ae2caa1b1f6533a1e23a613a18c9c67e5 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-10T07:36:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: pgmap v42: 161 pgs: 52 peering, 109 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: Health check failed: Reduced data availability: 8 pgs inactive, 50 pgs peering (PG_AVAILABILITY) 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: osdmap e107: 8 total, 7 up, 8 in 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:31.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:31 smithi136 ceph-mon[57888]: failed to encode map e107 with expected crc 2024-02-10T07:36:32.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:31 smithi064 podman[70075]: 2024-02-10 07:36:31.741037628 +0000 UTC m=+0.617615139 container remove 4326fee4be41c7b74533a7109bc1632ae2caa1b1f6533a1e23a613a18c9c67e5 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T07:36:32.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:31 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T07:36:32.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:31 smithi064 systemd[1]: Stopped Ceph osd.0 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:36:32.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:31 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.0.service: Consumed 7.184s CPU time. 2024-02-10T07:36:32.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:31 smithi064 systemd[1]: Starting Ceph osd.0 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:36:32.502 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 podman[70178]: 2024-02-10 07:36:32.090376582 +0000 UTC m=+0.017256373 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:36:32.503 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 podman[70178]: 2024-02-10 07:36:32.191968511 +0000 UTC m=+0.118848340 container create 462cea3cf484265e6d088e1ccd2a8985c6d307301b0fee3208a12353fc323fef (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T07:36:32.503 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 podman[70178]: 2024-02-10 07:36:32.428328315 +0000 UTC m=+0.355208099 container init 462cea3cf484265e6d088e1ccd2a8985c6d307301b0fee3208a12353fc323fef (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T07:36:32.503 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 podman[70178]: 2024-02-10 07:36:32.431438863 +0000 UTC m=+0.358318649 container start 462cea3cf484265e6d088e1ccd2a8985c6d307301b0fee3208a12353fc323fef (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T07:36:32.503 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 podman[70178]: 2024-02-10 07:36:32.502516578 +0000 UTC m=+0.429396369 container attach 462cea3cf484265e6d088e1ccd2a8985c6d307301b0fee3208a12353fc323fef (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:36:33.207 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T07:36:33.207 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T07:36:33.207 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T07:36:33.207 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T07:36:33.208 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T07:36:33.208 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T07:36:33.208 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T07:36:33.208 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T07:36:33.208 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 bash[70178]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T07:36:33.208 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T07:36:33.209 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T07:36:33.209 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T07:36:33.209 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T07:36:33.209 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate[70194]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T07:36:33.209 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:32 smithi064 podman[70178]: 2024-02-10 07:36:32.903916092 +0000 UTC m=+0.830795877 container died 462cea3cf484265e6d088e1ccd2a8985c6d307301b0fee3208a12353fc323fef (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-10T07:36:33.539 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:33 smithi064 ceph-mon[64470]: pgmap v44: 161 pgs: 6 stale+active+clean, 52 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:36:33.540 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:33 smithi064 ceph-mon[67016]: pgmap v44: 161 pgs: 6 stale+active+clean, 52 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:36:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:33 smithi136 ceph-mon[57888]: pgmap v44: 161 pgs: 6 stale+active+clean, 52 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:36:33.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:33 smithi064 podman[70325]: 2024-02-10 07:36:33.539662601 +0000 UTC m=+0.630273155 container remove 462cea3cf484265e6d088e1ccd2a8985c6d307301b0fee3208a12353fc323fef (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:36:33.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:33 smithi064 podman[70389]: 2024-02-10 07:36:33.757077976 +0000 UTC m=+0.115062517 container create fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-10T07:36:33.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:33 smithi064 podman[70389]: 2024-02-10 07:36:33.659985744 +0000 UTC m=+0.017970291 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:36:34.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:34 smithi064 podman[70389]: 2024-02-10 07:36:34.009328063 +0000 UTC m=+0.367312604 container init fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:36:34.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:34 smithi064 podman[70389]: 2024-02-10 07:36:34.012275357 +0000 UTC m=+0.370259899 container start fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:36:34.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:34 smithi064 ceph-osd[70408]: -- 172.21.15.64:0/1018731315 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56286d6eb1e0 con 0x56286d6bcc00 2024-02-10T07:36:34.314 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:34 smithi064 bash[70389]: fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b 2024-02-10T07:36:34.315 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:34 smithi064 systemd[1]: Started Ceph osd.0 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:36:34.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:36:34.562+0000 7f26c88e27c0 -1 Falling back to public interface 2024-02-10T07:36:35.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:35 smithi064 ceph-mon[67016]: pgmap v45: 161 pgs: 6 stale+active+clean, 52 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:36:35.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:35 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:35.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:35 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:35.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:35 smithi064 ceph-mon[64470]: pgmap v45: 161 pgs: 6 stale+active+clean, 52 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:36:35.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:35 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:35.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:35 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:35.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:35 smithi136 ceph-mon[57888]: pgmap v45: 161 pgs: 6 stale+active+clean, 52 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:36:35.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:35 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:35.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:35 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:36.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:36 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:36.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:36 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:36 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:37.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-mon[64470]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-10T07:36:37.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:37.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-mon[64470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 50 pgs peering) 2024-02-10T07:36:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-mon[67016]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-10T07:36:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:37.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-mon[67016]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 50 pgs peering) 2024-02-10T07:36:37.421 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:36:37.276+0000 7f26c88e27c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-10T07:36:37.421 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:36:37.309+0000 7f26c88e27c0 -1 osd.0 105 log_to_monitors true 2024-02-10T07:36:37.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:37 smithi136 ceph-mon[57888]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-10T07:36:37.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:37 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:37.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:37 smithi136 ceph-mon[57888]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 50 pgs peering) 2024-02-10T07:36:37.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:36:37] "GET /metrics HTTP/1.1" 200 38197 "" "Prometheus/2.33.4" 2024-02-10T07:36:38.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:38 smithi136 ceph-mon[57888]: from='osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:36:38.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:38 smithi136 ceph-mon[57888]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:36:38.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:38 smithi064 ceph-mon[64470]: from='osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:36:38.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:38 smithi064 ceph-mon[64470]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:36:38.564 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:36:38 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:36:38.205+0000 7f26b9d3a700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:36:38.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:38 smithi064 ceph-mon[67016]: from='osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:36:38.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:38 smithi064 ceph-mon[67016]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:36:38.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:36:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:36:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 377 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: osdmap e108: 8 total, 7 up, 8 in 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: from='osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:39.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:39.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:39 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 377 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-10T07:36:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:36:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: osdmap e108: 8 total, 7 up, 8 in 2024-02-10T07:36:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: from='osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 377 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: osdmap e108: 8 total, 7 up, 8 in 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: from='osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:39.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:39 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: failed to encode map e108 with expected crc 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984] boot 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: failed to encode map e108 with expected crc 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984] boot 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: failed to encode map e108 with expected crc 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: osd.0 [v2:172.21.15.64:6802/3651359984,v1:172.21.15.64:6803/3651359984] boot 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: failed to encode map e109 with expected crc 2024-02-10T07:36:40.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:40.523 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: pgmap v50: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 29/627 objects degraded (4.625%) 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:41 smithi136 ceph-mon[57888]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: pgmap v50: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 29/627 objects degraded (4.625%) 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[64470]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: pgmap v50: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 29/627 objects degraded (4.625%) 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:41.536 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:41 smithi064 ceph-mon[67016]: failed to encode map e110 with expected crc 2024-02-10T07:36:43.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:43 smithi136 ceph-mon[57888]: pgmap v52: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-02-10T07:36:43.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:43.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:36:43.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:36:43.299+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:36:43.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[64470]: pgmap v52: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[67016]: pgmap v52: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:36:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:44.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:44.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:44.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:44.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:44.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:45.466 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:45 smithi064 ceph-mon[67016]: pgmap v53: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:45.466 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:45 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-02-10T07:36:45.466 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:45 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:36:45.466 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:45 smithi064 ceph-mon[64470]: pgmap v53: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:45.466 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:45 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-02-10T07:36:45.466 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:45 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:45 smithi136 ceph-mon[57888]: pgmap v53: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:45 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-02-10T07:36:45.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:45 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:36:46.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:46 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:46.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:46 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:46.483 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:46 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:47.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:47 smithi136 ceph-mon[57888]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:36:47.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:47 smithi064 ceph-mon[67016]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:36:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:47 smithi064 ceph-mon[64470]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:36:47.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:36:47] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.33.4" 2024-02-10T07:36:48.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:36:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:36:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:36:49.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:49 smithi064 ceph-mon[64470]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:49.513 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:49 smithi064 ceph-mon[67016]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:49.513 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:36:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:49 smithi136 ceph-mon[57888]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (3m) 11s ago 8m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (2m) 59s ago 7m 47.3M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (2m) 11s ago 7m 53.9M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (2m) 59s ago 12m 448M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (2m) 11s ago 14m 502M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (107s) 11s ago 14m 45.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:36:50.179 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (65s) 59s ago 12m 20.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (81s) 11s ago 13m 37.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (3m) 11s ago 8m 16.5M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (2m) 59s ago 8m 17.3M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (16s) 11s ago 12m 13.5M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (11m) 11s ago 11m 56.7M 2736M 17.2.0 e1d6a67b021e 5abe808502d7 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (11m) 11s ago 11m 51.7M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (11m) 11s ago 11m 56.4M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (10m) 59s ago 10m 53.1M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (10m) 59s ago 10m 49.9M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (10m) 59s ago 10m 50.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (9m) 59s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (2m) 59s ago 8m 54.8M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (7m) 11s ago 7m 73.7M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (7m) 59s ago 7m 72.4M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:36:50.180 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (7m) 11s ago 7m 72.6M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:36:50.181 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (7m) 59s ago 7m 71.9M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:36:50.923 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "6/23 daemons upgraded", 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:36:50.924 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:36:51.324 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:36:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[64470]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:36:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[64470]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[64470]: from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[64470]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2243853178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[67016]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:36:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[67016]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[67016]: from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[67016]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:51 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2243853178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:51 smithi136 ceph-mon[57888]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:36:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:51 smithi136 ceph-mon[57888]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:51 smithi136 ceph-mon[57888]: from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:51 smithi136 ceph-mon[57888]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:51 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2243853178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:36:52.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:52 smithi064 ceph-mon[64470]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:52.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:52 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/616925712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:36:52.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:52 smithi064 ceph-mon[67016]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:52.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:52 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/616925712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:36:52.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:52 smithi136 ceph-mon[57888]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:36:52.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:52 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/616925712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:36:53.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:53 smithi064 ceph-mon[64470]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-02-10T07:36:53.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:53 smithi064 ceph-mon[67016]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-02-10T07:36:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:53 smithi136 ceph-mon[57888]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-02-10T07:36:54.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:54 smithi136 ceph-mon[57888]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:54.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:54 smithi064 ceph-mon[64470]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:54.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:54 smithi064 ceph-mon[67016]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:54.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:55.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:55 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:55 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:55 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:56 smithi136 ceph-mon[57888]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:56 smithi064 ceph-mon[64470]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:56.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:56 smithi064 ceph-mon[67016]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:36:57.736 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:36:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:36:57] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.33.4" 2024-02-10T07:36:58.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:36:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:36:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[64470]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[64470]: Upgrade: osd.1 is safe to restart 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:59.554 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[67016]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:59.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:59.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:59.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[67016]: Upgrade: osd.1 is safe to restart 2024-02-10T07:36:59.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:59.555 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:36:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:59 smithi136 ceph-mon[57888]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:36:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:59 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T07:36:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:59 smithi136 ceph-mon[57888]: Upgrade: osd.1 is safe to restart 2024-02-10T07:36:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:36:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:36:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[64470]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[64470]: Upgrade: Updating osd.1 2024-02-10T07:37:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:37:01.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[64470]: Deploying daemon osd.1 on smithi064 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[67016]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[67016]: Upgrade: Updating osd.1 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:01.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:00 smithi064 ceph-mon[67016]: Deploying daemon osd.1 on smithi064 2024-02-10T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:00 smithi136 ceph-mon[57888]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:00 smithi136 ceph-mon[57888]: Upgrade: Updating osd.1 2024-02-10T07:37:01.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:01.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:37:01.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:01.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:00 smithi136 ceph-mon[57888]: Deploying daemon osd.1 on smithi064 2024-02-10T07:37:03.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:02 smithi064 ceph-mon[64470]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:03.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:02 smithi064 ceph-mon[67016]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:03.315 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:03 smithi064 systemd[1]: Stopping Ceph osd.1 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:37:03.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:02 smithi136 ceph-mon[57888]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:03.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[35287]: 2024-02-10T07:37:03.554+0000 7f622c7be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:37:03.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[35287]: 2024-02-10T07:37:03.554+0000 7f622c7be700 -1 osd.1 110 *** Got signal Terminated *** 2024-02-10T07:37:03.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[35287]: 2024-02-10T07:37:03.554+0000 7f622c7be700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:37:04.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:04 smithi064 ceph-mon[64470]: osd.1 marked itself down and dead 2024-02-10T07:37:04.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:04 smithi064 ceph-mon[67016]: osd.1 marked itself down and dead 2024-02-10T07:37:04.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:04 smithi136 ceph-mon[57888]: osd.1 marked itself down and dead 2024-02-10T07:37:04.998 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:04 smithi064 podman[72415]: 2024-02-10 07:37:04.737720157 +0000 UTC m=+1.337506880 container died 5abe808502d7bf7ecbbaba2d2e20929c3b5eeac152607640a030e51bd97a9662 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T07:37:05.314 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:05 smithi064 podman[72415]: 2024-02-10 07:37:05.086986183 +0000 UTC m=+1.686772917 container cleanup 5abe808502d7bf7ecbbaba2d2e20929c3b5eeac152607640a030e51bd97a9662 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, distribution-scope=public, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T07:37:05.314 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:05 smithi064 bash[72415]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: osdmap e111: 8 total, 7 up, 8 in 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[64470]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: osdmap e111: 8 total, 7 up, 8 in 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.316 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:05 smithi064 ceph-mon[67016]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:05.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: osdmap e111: 8 total, 7 up, 8 in 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:05 smithi136 ceph-mon[57888]: failed to encode map e111 with expected crc 2024-02-10T07:37:05.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:05 smithi064 podman[72434]: 2024-02-10 07:37:05.355204946 +0000 UTC m=+0.612203345 container remove 5abe808502d7bf7ecbbaba2d2e20929c3b5eeac152607640a030e51bd97a9662 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, release=754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True) 2024-02-10T07:37:06.108 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:05 smithi064 podman[72525]: 2024-02-10 07:37:05.755086759 +0000 UTC m=+0.017475316 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:37:06.108 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:05 smithi064 podman[72525]: 2024-02-10 07:37:05.989337498 +0000 UTC m=+0.251726090 container create d7bf4d597dbd6e1c915de6385e2b7df190f322d16690ed45e5ceb43db8960986 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-10T07:37:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: osdmap e112: 8 total, 7 up, 8 in 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[64470]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: osdmap e112: 8 total, 7 up, 8 in 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:06 smithi064 ceph-mon[67016]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.566 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:06 smithi064 podman[72525]: 2024-02-10 07:37:06.258466492 +0000 UTC m=+0.520855043 container init d7bf4d597dbd6e1c915de6385e2b7df190f322d16690ed45e5ceb43db8960986 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T07:37:06.566 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:06 smithi064 podman[72525]: 2024-02-10 07:37:06.261770129 +0000 UTC m=+0.524158679 container start d7bf4d597dbd6e1c915de6385e2b7df190f322d16690ed45e5ceb43db8960986 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:37:06.566 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:06 smithi064 podman[72525]: 2024-02-10 07:37:06.358438476 +0000 UTC m=+0.620827026 container attach d7bf4d597dbd6e1c915de6385e2b7df190f322d16690ed45e5ceb43db8960986 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-10T07:37:06.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: osdmap e112: 8 total, 7 up, 8 in 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:06 smithi136 ceph-mon[57888]: failed to encode map e112 with expected crc 2024-02-10T07:37:06.911 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:06 smithi064 podman[72525]: 2024-02-10 07:37:06.565105329 +0000 UTC m=+0.827493892 container died d7bf4d597dbd6e1c915de6385e2b7df190f322d16690ed45e5ceb43db8960986 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T07:37:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:07 smithi064 ceph-mon[64470]: pgmap v66: 161 pgs: 40 peering, 121 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:37:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:07 smithi064 ceph-mon[64470]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T07:37:07.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:37:07] "GET /metrics HTTP/1.1" 200 38342 "" "Prometheus/2.33.4" 2024-02-10T07:37:07.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:07 smithi064 ceph-mon[67016]: pgmap v66: 161 pgs: 40 peering, 121 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:37:07.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:07 smithi064 ceph-mon[67016]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T07:37:07.565 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 podman[72551]: 2024-02-10 07:37:07.245381492 +0000 UTC m=+0.674524265 container remove d7bf4d597dbd6e1c915de6385e2b7df190f322d16690ed45e5ceb43db8960986 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T07:37:07.565 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-10T07:37:07.565 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 systemd[1]: Stopped Ceph osd.1 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:37:07.566 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.1.service: Consumed 7.573s CPU time. 2024-02-10T07:37:07.566 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 systemd[1]: Starting Ceph osd.1 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:37:07.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:07 smithi136 ceph-mon[57888]: pgmap v66: 161 pgs: 40 peering, 121 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:37:07.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:07 smithi136 ceph-mon[57888]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T07:37:07.955 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 podman[72656]: 2024-02-10 07:37:07.595879595 +0000 UTC m=+0.017597192 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:37:07.955 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 podman[72656]: 2024-02-10 07:37:07.713183238 +0000 UTC m=+0.134900884 container create 28f58f34cc74ad0f9fea6bda59155c4ee96449dd1f02d9a6f8f6ce2b3312f3e6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T07:37:07.955 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 podman[72656]: 2024-02-10 07:37:07.955607309 +0000 UTC m=+0.377324902 container init 28f58f34cc74ad0f9fea6bda59155c4ee96449dd1f02d9a6f8f6ce2b3312f3e6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T07:37:08.314 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:07 smithi064 podman[72656]: 2024-02-10 07:37:07.960407392 +0000 UTC m=+0.382124974 container start 28f58f34cc74ad0f9fea6bda59155c4ee96449dd1f02d9a6f8f6ce2b3312f3e6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:37:08.314 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 podman[72656]: 2024-02-10 07:37:08.063935311 +0000 UTC m=+0.485652912 container attach 28f58f34cc74ad0f9fea6bda59155c4ee96449dd1f02d9a6f8f6ce2b3312f3e6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-10T07:37:08.715 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 bash[72656]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T07:37:08.716 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate[72670]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T07:37:08.717 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:08 smithi064 podman[72656]: 2024-02-10 07:37:08.435682535 +0000 UTC m=+0.857400132 container died 28f58f34cc74ad0f9fea6bda59155c4ee96449dd1f02d9a6f8f6ce2b3312f3e6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T07:37:08.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:37:09.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:09 smithi064 ceph-mon[64470]: pgmap v67: 161 pgs: 40 peering, 121 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:37:09.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:09 smithi064 ceph-mon[67016]: pgmap v67: 161 pgs: 40 peering, 121 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:37:09.315 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 podman[72808]: 2024-02-10 07:37:09.017323267 +0000 UTC m=+0.576460828 container remove 28f58f34cc74ad0f9fea6bda59155c4ee96449dd1f02d9a6f8f6ce2b3312f3e6 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:37:09.315 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 podman[72869]: 2024-02-10 07:37:09.131121772 +0000 UTC m=+0.017878337 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:37:09.315 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 podman[72869]: 2024-02-10 07:37:09.242913528 +0000 UTC m=+0.129670085 container create 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-02-10T07:37:09.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:09 smithi136 ceph-mon[57888]: pgmap v67: 161 pgs: 40 peering, 121 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:37:09.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 podman[72869]: 2024-02-10 07:37:09.487366732 +0000 UTC m=+0.374123292 container init 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0) 2024-02-10T07:37:09.815 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 podman[72869]: 2024-02-10 07:37:09.490321322 +0000 UTC m=+0.377077879 container start 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-10T07:37:09.815 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 ceph-osd[72888]: -- 172.21.15.64:0/1411287726 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b851ced1e0 con 0x55b851cbec00 2024-02-10T07:37:09.815 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 bash[72869]: 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 2024-02-10T07:37:09.815 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:09 smithi064 systemd[1]: Started Ceph osd.1 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:37:10.314 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:37:10.030+0000 7fb7891737c0 -1 Falling back to public interface 2024-02-10T07:37:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[64470]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:37:10.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[64470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-10T07:37:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[67016]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:37:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:10 smithi064 ceph-mon[67016]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-10T07:37:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:10 smithi136 ceph-mon[57888]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:10.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:10 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:37:10.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:10 smithi136 ceph-mon[57888]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-10T07:37:12.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:12 smithi064 ceph-mon[64470]: pgmap v69: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:12.882 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:12 smithi064 ceph-mon[67016]: pgmap v69: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:12.882 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:37:12.747+0000 7fb7891737c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-10T07:37:12.882 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:12 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:37:12.775+0000 7fb7891737c0 -1 osd.1 110 log_to_monitors true 2024-02-10T07:37:12.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:12 smithi136 ceph-mon[57888]: pgmap v69: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:13.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-mon[67016]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:37:13.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:13.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:13.815 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:37:13.651+0000 7fb77a5cb700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:37:13.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-mon[64470]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:37:13.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:13.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:13 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:13.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:13 smithi136 ceph-mon[57888]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:37:13.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:13 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:13.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:13 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: pgmap v70: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:37:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: osdmap e113: 8 total, 7 up, 8 in 2024-02-10T07:37:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: pgmap v70: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: osdmap e113: 8 total, 7 up, 8 in 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:14.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: pgmap v70: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: osdmap e113: 8 total, 7 up, 8 in 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: from='osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: failed to encode map e113 with expected crc 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:14.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.899 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:15.899 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596] boot 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[64470]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596] boot 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.901 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:15 smithi064 ceph-mon[67016]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T07:37:15.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: osd.1 [v2:172.21.15.64:6810/2916438596,v1:172.21.15.64:6811/2916438596] boot 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: failed to encode map e114 with expected crc 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:15.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:15 smithi136 ceph-mon[57888]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:16.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:16 smithi136 ceph-mon[57888]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:17.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.028 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[67016]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 86/627 objects degraded (13.716%) 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.029 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.030 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:16 smithi064 ceph-mon[64470]: failed to encode map e115 with expected crc 2024-02-10T07:37:17.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:37:17] "GET /metrics HTTP/1.1" 200 38412 "" "Prometheus/2.33.4" 2024-02-10T07:37:18.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:18.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-10T07:37:18.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:37:19.064 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:18 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:37:18.827+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T07:37:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: pgmap v75: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 86/627 objects degraded (13.716%) 2024-02-10T07:37:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: pgmap v75: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 86/627 objects degraded (13.716%) 2024-02-10T07:37:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:19.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: pgmap v75: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 86/627 objects degraded (13.716%) 2024-02-10T07:37:19.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:20 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:20 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T07:37:20.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:20 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:20.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:20 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T07:37:20.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:20 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:20.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:20 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T07:37:21.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:21 smithi064 ceph-mon[64470]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:37:21.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:21 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded) 2024-02-10T07:37:21.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:21 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:37:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:21 smithi064 ceph-mon[67016]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:37:21.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:21 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded) 2024-02-10T07:37:21.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:21 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:37:21.692 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:37:21.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:21 smithi136 ceph-mon[57888]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:37:21.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:21 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/627 objects degraded (13.716%), 22 pgs degraded) 2024-02-10T07:37:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:21 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (3m) 8s ago 8m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (3m) 91s ago 8m 47.3M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (2m) 8s ago 7m 54.1M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (2m) 91s ago 13m 448M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (3m) 8s ago 14m 503M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (2m) 8s ago 15m 47.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (97s) 91s ago 13m 20.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (113s) 8s ago 13m 38.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (3m) 8s ago 9m 16.2M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (3m) 91s ago 9m 17.3M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:37:22.357 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (48s) 8s ago 12m 42.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (13s) 8s ago 12m 13.5M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (12m) 8s ago 12m 53.6M 2736M 17.2.0 e1d6a67b021e e580bd1ad888 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (11m) 8s ago 11m 57.4M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (11m) 91s ago 11m 53.1M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (10m) 91s ago 10m 49.9M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (10m) 91s ago 10m 50.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (10m) 91s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (2m) 91s ago 9m 54.8M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (7m) 8s ago 7m 74.0M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (7m) 91s ago 7m 72.4M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (7m) 8s ago 7m 72.7M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:37:22.358 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (7m) 91s ago 7m 71.9M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:37:22.759 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:37:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:37:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:37:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T07:37:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T07:37:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:37:22.761 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "7/23 daemons upgraded", 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:37:23.100 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:37:23.507 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[64470]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[64470]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[64470]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/4186050720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[67016]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[67016]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:23.508 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[67016]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:23.509 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:23 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/4186050720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:23.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:23 smithi136 ceph-mon[57888]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:37:23.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:23 smithi136 ceph-mon[57888]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:23.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:23 smithi136 ceph-mon[57888]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:23.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:23 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/4186050720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:24.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:24 smithi064 ceph-mon[64470]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:24 smithi064 ceph-mon[64470]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:24.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:24 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3411895654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:37:24.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:24 smithi064 ceph-mon[67016]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:24.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:24 smithi064 ceph-mon[67016]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:24.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:24 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/3411895654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:37:24.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:24 smithi136 ceph-mon[57888]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:24.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:24 smithi136 ceph-mon[57888]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:24 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/3411895654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:37:25.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:25 smithi064 ceph-mon[64470]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-10T07:37:25.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:25 smithi064 ceph-mon[67016]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-10T07:37:25.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:25 smithi136 ceph-mon[57888]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-10T07:37:26.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:26 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:26.557 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:26 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:26 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:27.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:27 smithi064 ceph-mon[64470]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:37:27.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:27 smithi064 ceph-mon[67016]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:37:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:27 smithi136 ceph-mon[57888]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:37:27.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:37:27] "GET /metrics HTTP/1.1" 200 38412 "" "Prometheus/2.33.4" 2024-02-10T07:37:28.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:28.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-10T07:37:28.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:37:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[67016]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-10T07:37:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[64470]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-10T07:37:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:29 smithi136 ceph-mon[57888]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-10T07:37:29.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:29.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:29.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:31.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:31 smithi064 ceph-mon[64470]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:31.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:31 smithi064 ceph-mon[67016]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:31 smithi136 ceph-mon[57888]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:33 smithi064 ceph-mon[64470]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:33.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:33 smithi064 ceph-mon[67016]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:33 smithi136 ceph-mon[57888]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:34.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:34 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:34.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:34 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:34.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:34 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:35.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:35 smithi064 ceph-mon[64470]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:35.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:35 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:35.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:35 smithi064 ceph-mon[64470]: Upgrade: osd.2 is safe to restart 2024-02-10T07:37:35.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:35 smithi064 ceph-mon[67016]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:35.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:35 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:35.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:35 smithi064 ceph-mon[67016]: Upgrade: osd.2 is safe to restart 2024-02-10T07:37:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:35 smithi136 ceph-mon[57888]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:35 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T07:37:35.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:35 smithi136 ceph-mon[57888]: Upgrade: osd.2 is safe to restart 2024-02-10T07:37:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:37:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:36.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:37:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:36 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:36.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:36 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:36.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:36 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:36 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:37:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:36 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:37.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-mon[64470]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:37.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-mon[64470]: Upgrade: Updating osd.2 2024-02-10T07:37:37.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-mon[64470]: Deploying daemon osd.2 on smithi064 2024-02-10T07:37:37.405 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-mon[67016]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:37.405 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-mon[67016]: Upgrade: Updating osd.2 2024-02-10T07:37:37.406 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-mon[67016]: Deploying daemon osd.2 on smithi064 2024-02-10T07:37:37.662 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:37:37] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-02-10T07:37:37.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:37 smithi136 ceph-mon[57888]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:37.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:37 smithi136 ceph-mon[57888]: Upgrade: Updating osd.2 2024-02-10T07:37:37.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:37 smithi136 ceph-mon[57888]: Deploying daemon osd.2 on smithi064 2024-02-10T07:37:38.814 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:38 smithi064 systemd[1]: Stopping Ceph osd.2 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:37:38.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:37:39.294 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:38 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[38216]: 2024-02-10T07:37:38.945+0000 7f2a7749b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:37:39.294 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:38 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[38216]: 2024-02-10T07:37:38.945+0000 7f2a7749b700 -1 osd.2 115 *** Got signal Terminated *** 2024-02-10T07:37:39.295 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:38 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[38216]: 2024-02-10T07:37:38.945+0000 7f2a7749b700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:37:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:39 smithi064 ceph-mon[64470]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:39 smithi064 ceph-mon[64470]: osd.2 marked itself down and dead 2024-02-10T07:37:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:39 smithi064 ceph-mon[67016]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:39 smithi064 ceph-mon[67016]: osd.2 marked itself down and dead 2024-02-10T07:37:39.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:39 smithi136 ceph-mon[57888]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:39 smithi136 ceph-mon[57888]: osd.2 marked itself down and dead 2024-02-10T07:37:40.064 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:39 smithi064 podman[74884]: 2024-02-10 07:37:39.787956122 +0000 UTC m=+0.965177493 container died e580bd1ad8885b00a8a50ab63b77627c08455265e07e487502331c81d61da85f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.expose-services=, vcs-type=git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_CLEAN=True, distribution-scope=public, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, release=754, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.430 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[67016]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:40 smithi064 ceph-mon[64470]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.431 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 podman[74884]: 2024-02-10 07:37:40.153755576 +0000 UTC m=+1.330976940 container cleanup e580bd1ad8885b00a8a50ab63b77627c08455265e07e487502331c81d61da85f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, release=754) 2024-02-10T07:37:40.431 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 bash[74884]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2 2024-02-10T07:37:40.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:40 smithi136 ceph-mon[57888]: failed to encode map e116 with expected crc 2024-02-10T07:37:40.756 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 podman[74911]: 2024-02-10 07:37:40.430308044 +0000 UTC m=+0.642337784 container remove e580bd1ad8885b00a8a50ab63b77627c08455265e07e487502331c81d61da85f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8) 2024-02-10T07:37:41.024 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 podman[75003]: 2024-02-10 07:37:40.756202511 +0000 UTC m=+0.101251460 container create fa543e4f93a7b3dc7138a9c3c0119510014ca594eb7490b646dc32ef8470c896 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:37:41.025 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 podman[75003]: 2024-02-10 07:37:40.672383487 +0000 UTC m=+0.017432396 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:37:41.025 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 podman[75003]: 2024-02-10 07:37:40.950108863 +0000 UTC m=+0.295157767 container init fa543e4f93a7b3dc7138a9c3c0119510014ca594eb7490b646dc32ef8470c896 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T07:37:41.025 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:40 smithi064 podman[75003]: 2024-02-10 07:37:40.953253085 +0000 UTC m=+0.298301993 container start fa543e4f93a7b3dc7138a9c3c0119510014ca594eb7490b646dc32ef8470c896 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:37:41.025 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:41 smithi064 podman[75003]: 2024-02-10 07:37:41.024964186 +0000 UTC m=+0.370013090 container attach fa543e4f93a7b3dc7138a9c3c0119510014ca594eb7490b646dc32ef8470c896 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T07:37:41.298 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:41 smithi064 podman[75003]: 2024-02-10 07:37:41.258279792 +0000 UTC m=+0.603328702 container died fa543e4f93a7b3dc7138a9c3c0119510014ca594eb7490b646dc32ef8470c896 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:37:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: pgmap v87: 161 pgs: 26 peering, 135 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:37:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T07:37:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: osdmap e117: 8 total, 7 up, 8 in 2024-02-10T07:37:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[64470]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: pgmap v87: 161 pgs: 26 peering, 135 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: osdmap e117: 8 total, 7 up, 8 in 2024-02-10T07:37:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:41 smithi064 ceph-mon[67016]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: pgmap v87: 161 pgs: 26 peering, 135 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: osdmap e117: 8 total, 7 up, 8 in 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:41 smithi136 ceph-mon[57888]: failed to encode map e117 with expected crc 2024-02-10T07:37:42.314 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 podman[75030]: 2024-02-10 07:37:42.003615366 +0000 UTC m=+0.739881251 container remove fa543e4f93a7b3dc7138a9c3c0119510014ca594eb7490b646dc32ef8470c896 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-10T07:37:42.314 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T07:37:42.314 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 systemd[1]: Stopped Ceph osd.2 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:37:42.314 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.2.service: Consumed 6.282s CPU time. 2024-02-10T07:37:42.314 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 systemd[1]: Starting Ceph osd.2 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:37:42.740 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 podman[75131]: 2024-02-10 07:37:42.354627588 +0000 UTC m=+0.018438305 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:37:42.740 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 podman[75131]: 2024-02-10 07:37:42.454347511 +0000 UTC m=+0.118158265 container create 67b0dbed668385dcf9bbd5b5238cbdc6ad0e96395d496f2c76590ffd54bd47f8 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate, RELEASE=main-290212a, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T07:37:42.740 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 podman[75131]: 2024-02-10 07:37:42.674529848 +0000 UTC m=+0.338340561 container init 67b0dbed668385dcf9bbd5b5238cbdc6ad0e96395d496f2c76590ffd54bd47f8 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T07:37:42.740 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 podman[75131]: 2024-02-10 07:37:42.677858237 +0000 UTC m=+0.341668947 container start 67b0dbed668385dcf9bbd5b5238cbdc6ad0e96395d496f2c76590ffd54bd47f8 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T07:37:42.740 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:42 smithi064 podman[75131]: 2024-02-10 07:37:42.74035519 +0000 UTC m=+0.404165900 container attach 67b0dbed668385dcf9bbd5b5238cbdc6ad0e96395d496f2c76590ffd54bd47f8 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T07:37:43.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-mon[64470]: pgmap v89: 161 pgs: 10 stale+active+clean, 26 peering, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:37:43.477 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-mon[67016]: pgmap v89: 161 pgs: 10 stale+active+clean, 26 peering, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T07:37:43.477 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 bash[75131]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate[75147]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T07:37:43.478 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 podman[75131]: 2024-02-10 07:37:43.145786629 +0000 UTC m=+0.809597340 container died 67b0dbed668385dcf9bbd5b5238cbdc6ad0e96395d496f2c76590ffd54bd47f8 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T07:37:43.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:43 smithi136 ceph-mon[57888]: pgmap v89: 161 pgs: 10 stale+active+clean, 26 peering, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:37:44.064 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 podman[75278]: 2024-02-10 07:37:43.752534617 +0000 UTC m=+0.597950061 container remove 67b0dbed668385dcf9bbd5b5238cbdc6ad0e96395d496f2c76590ffd54bd47f8 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T07:37:44.064 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 podman[75345]: 2024-02-10 07:37:43.866264731 +0000 UTC m=+0.018311099 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:37:44.064 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:43 smithi064 podman[75345]: 2024-02-10 07:37:43.978385759 +0000 UTC m=+0.130432161 container create 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T07:37:44.428 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:44 smithi064 podman[75345]: 2024-02-10 07:37:44.197314097 +0000 UTC m=+0.349360456 container init 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:37:44.428 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:44 smithi064 podman[75345]: 2024-02-10 07:37:44.200243689 +0000 UTC m=+0.352290048 container start 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T07:37:44.428 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-osd[75363]: -- 172.21.15.64:0/4236769527 <== mon.1 v2:172.21.15.64:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f546ab1e0 con 0x561f5467cc00 2024-02-10T07:37:44.428 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:44 smithi064 bash[75345]: 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e 2024-02-10T07:37:44.428 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:44 smithi064 systemd[1]: Started Ceph osd.2 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:37:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:44 smithi136 ceph-mon[57888]: pgmap v90: 161 pgs: 8 stale+active+clean, 1 active+undersized+degraded, 26 peering, 6 active+undersized, 120 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 5/627 objects degraded (0.797%) 2024-02-10T07:37:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:44 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:37:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[64470]: pgmap v90: 161 pgs: 8 stale+active+clean, 1 active+undersized+degraded, 26 peering, 6 active+undersized, 120 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 5/627 objects degraded (0.797%) 2024-02-10T07:37:44.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[67016]: pgmap v90: 161 pgs: 8 stale+active+clean, 1 active+undersized+degraded, 26 peering, 6 active+undersized, 120 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 5/627 objects degraded (0.797%) 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:44.745 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:45.064 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:37:44.743+0000 7f2ed38587c0 -1 Falling back to public interface 2024-02-10T07:37:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:45 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:45.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:45 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:45 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:46.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:46 smithi136 ceph-mon[57888]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:46.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:46 smithi136 ceph-mon[57888]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T07:37:46.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:46 smithi064 ceph-mon[64470]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:46.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:46 smithi064 ceph-mon[64470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T07:37:46.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:46 smithi064 ceph-mon[67016]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:46.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:46 smithi064 ceph-mon[67016]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T07:37:47.758 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:37:47] "GET /metrics HTTP/1.1" 200 38494 "" "Prometheus/2.33.4" 2024-02-10T07:37:47.759 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:37:47.461+0000 7f2ed38587c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-10T07:37:47.759 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:37:47.486+0000 7f2ed38587c0 -1 osd.2 115 log_to_monitors true 2024-02-10T07:37:48.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:48 smithi136 ceph-mon[57888]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:48.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:48 smithi136 ceph-mon[57888]: from='osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:37:48.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:48 smithi136 ceph-mon[57888]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:37:48.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:48.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-10T07:37:48.723 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:37:48.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-mon[67016]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:48.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-mon[67016]: from='osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:37:48.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-mon[67016]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:37:48.814 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:37:48.471+0000 7f2ec4cb0700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:37:48.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-mon[64470]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:48.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-mon[64470]: from='osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:37:48.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:48 smithi064 ceph-mon[64470]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:37:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: osdmap e118: 8 total, 7 up, 8 in 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: from='osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: osdmap e118: 8 total, 7 up, 8 in 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: from='osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:49.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: osdmap e118: 8 total, 7 up, 8 in 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: from='osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: failed to encode map e118 with expected crc 2024-02-10T07:37:49.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:49.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361] boot 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:37:50.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:50.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:50 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:50.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:50.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:37:50.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361] boot 2024-02-10T07:37:50.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T07:37:50.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: osd.2 [v2:172.21.15.64:6818/388380361,v1:172.21.15.64:6819/388380361] boot 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: failed to encode map e119 with expected crc 2024-02-10T07:37:50.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:37:50.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:50.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:50 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:51 smithi136 ceph-mon[57888]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[64470]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.752 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:51.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:51 smithi064 ceph-mon[67016]: failed to encode map e120 with expected crc 2024-02-10T07:37:52.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:52 smithi136 ceph-mon[57888]: pgmap v97: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-02-10T07:37:52.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:52 smithi064 ceph-mon[64470]: pgmap v97: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-02-10T07:37:52.790 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:52 smithi064 ceph-mon[67016]: pgmap v97: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-02-10T07:37:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:53 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T07:37:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:53 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:53 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:53 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:53 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:53 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:37:53.584+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:37:53.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T07:37:53.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:53.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:53 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:53.871 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:37:54.539 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:37:54.539 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (4m) 5s ago 9m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (3m) 2m ago 8m 47.3M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (3m) 5s ago 8m 54.2M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (3m) 2m ago 13m 448M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (3m) 5s ago 15m 506M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (2m) 5s ago 15m 47.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (2m) 2m ago 13m 20.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (2m) 5s ago 14m 38.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:37:54.540 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (4m) 5s ago 9m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (4m) 2m ago 9m 17.3M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (80s) 5s ago 13m 42.6M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (45s) 5s ago 13m 43.5M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (10s) 5s ago 12m 12.1M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (12m) 5s ago 12m 59.2M 2736M 17.2.0 e1d6a67b021e e65ca3100dcf 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (11m) 2m ago 11m 53.1M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (11m) 2m ago 11m 49.9M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (11m) 2m ago 11m 50.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (10m) 2m ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (3m) 2m ago 10m 54.8M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (8m) 5s ago 8m 74.2M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (8m) 2m ago 8m 72.4M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (8m) 5s ago 8m 72.9M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:37:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (8m) 2m ago 8m 71.9M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:37:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: pgmap v98: 161 pgs: 13 active+undersized+degraded, 28 active+undersized, 120 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-02-10T07:37:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[64470]: from='client.44316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: pgmap v98: 161 pgs: 13 active+undersized+degraded, 28 active+undersized, 120 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:37:54.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:37:54.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:37:54.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:54.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:54 smithi064 ceph-mon[67016]: from='client.44316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:37:54.943 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:37:54.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: pgmap v98: 161 pgs: 13 active+undersized+degraded, 28 active+undersized, 120 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-02-10T07:37:54.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:54.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:54.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:54.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:37:54.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:37:54.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:37:54.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:54.975 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:54 smithi136 ceph-mon[57888]: from='client.44316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:55.279 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:37:55.279 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:37:55.279 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "8/23 daemons upgraded", 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:37:55.280 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[64470]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1716797763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[64470]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[67016]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1716797763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:55.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:55 smithi064 ceph-mon[67016]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:55.691 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:37:55.977 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:55 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:55.977 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:55 smithi136 ceph-mon[57888]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:55.977 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:55 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1716797763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:37:55.977 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:55 smithi136 ceph-mon[57888]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:37:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[64470]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:37:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/627 objects degraded (8.134%), 13 pgs degraded) 2024-02-10T07:37:56.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:37:56.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/987201463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:37:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[67016]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:37:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/627 objects degraded (8.134%), 13 pgs degraded) 2024-02-10T07:37:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:37:56.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:56 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/987201463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:37:56.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:56 smithi136 ceph-mon[57888]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:37:56.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:56 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/627 objects degraded (8.134%), 13 pgs degraded) 2024-02-10T07:37:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:56 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:37:56.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:56 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/987201463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:37:57.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:37:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:37:57] "GET /metrics HTTP/1.1" 200 38494 "" "Prometheus/2.33.4" 2024-02-10T07:37:58.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-10T07:37:58.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:37:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:37:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:37:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:59 smithi064 ceph-mon[64470]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:37:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:37:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:59 smithi064 ceph-mon[67016]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:37:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:37:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:59 smithi136 ceph-mon[57888]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:37:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:37:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:37:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:01 smithi064 ceph-mon[64470]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:38:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:01 smithi064 ceph-mon[67016]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:38:01.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:01 smithi136 ceph-mon[57888]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:38:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:03 smithi064 ceph-mon[64470]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-02-10T07:38:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:03 smithi064 ceph-mon[67016]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-02-10T07:38:03.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:03 smithi136 ceph-mon[57888]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-02-10T07:38:04.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:04 smithi064 ceph-mon[67016]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:04.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:04.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:04 smithi064 ceph-mon[64470]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:04.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:04.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:04 smithi136 ceph-mon[57888]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:04.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:05.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:05 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:05.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:05 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:05.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:05 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:06.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:06 smithi064 ceph-mon[64470]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:06.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:06 smithi064 ceph-mon[67016]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:06.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:06 smithi136 ceph-mon[57888]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:07.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:38:07] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.33.4" 2024-02-10T07:38:08.973 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:38:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:38:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:38:09.520 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:09 smithi064 ceph-mon[67016]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:09.521 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:38:09.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:09 smithi064 ceph-mon[64470]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:09.521 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:38:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:09 smithi136 ceph-mon[57888]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:38:10.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:38:10.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[67016]: Upgrade: osd.3 is safe to restart 2024-02-10T07:38:10.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:10.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:38:10.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:38:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[64470]: Upgrade: osd.3 is safe to restart 2024-02-10T07:38:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:38:10.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:10.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:10 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T07:38:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:10 smithi136 ceph-mon[57888]: Upgrade: osd.3 is safe to restart 2024-02-10T07:38:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:38:10.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:11.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:11 smithi064 ceph-mon[67016]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:11.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:11 smithi064 ceph-mon[67016]: Upgrade: Updating osd.3 2024-02-10T07:38:11.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:11 smithi064 ceph-mon[67016]: Deploying daemon osd.3 on smithi064 2024-02-10T07:38:11.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:11 smithi064 ceph-mon[64470]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:11.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:11 smithi064 ceph-mon[64470]: Upgrade: Updating osd.3 2024-02-10T07:38:11.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:11 smithi064 ceph-mon[64470]: Deploying daemon osd.3 on smithi064 2024-02-10T07:38:11.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:11 smithi136 ceph-mon[57888]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:11.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:11 smithi136 ceph-mon[57888]: Upgrade: Updating osd.3 2024-02-10T07:38:11.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:11 smithi136 ceph-mon[57888]: Deploying daemon osd.3 on smithi064 2024-02-10T07:38:13.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:13 smithi064 ceph-mon[64470]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:13.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:13 smithi064 ceph-mon[67016]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:13.565 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:13 smithi064 systemd[1]: Stopping Ceph osd.3 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:38:13.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:13 smithi136 ceph-mon[57888]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:14.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[41193]: 2024-02-10T07:38:13.663+0000 7f77767be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:38:14.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[41193]: 2024-02-10T07:38:13.663+0000 7f77767be700 -1 osd.3 120 *** Got signal Terminated *** 2024-02-10T07:38:14.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[41193]: 2024-02-10T07:38:13.663+0000 7f77767be700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:38:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:14 smithi064 ceph-mon[64470]: osd.3 marked itself down and dead 2024-02-10T07:38:14.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:14 smithi064 ceph-mon[67016]: osd.3 marked itself down and dead 2024-02-10T07:38:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:14 smithi136 ceph-mon[57888]: osd.3 marked itself down and dead 2024-02-10T07:38:14.998 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:14 smithi064 podman[77356]: 2024-02-10 07:38:14.728035686 +0000 UTC m=+1.173790715 container died e65ca3100dcf771afecbf4818de08775193ce3e69928ce1d855ebb37cdb4e2c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, release=754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-10T07:38:15.271 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77356]: 2024-02-10 07:38:15.097006227 +0000 UTC m=+1.542761261 container cleanup e65ca3100dcf771afecbf4818de08775193ce3e69928ce1d855ebb37cdb4e2c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_CLEAN=True, distribution-scope=public, release=754, name=centos-stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T07:38:15.271 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 bash[77356]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3 2024-02-10T07:38:15.564 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77374]: 2024-02-10 07:38:15.348167978 +0000 UTC m=+0.621746783 container remove e65ca3100dcf771afecbf4818de08775193ce3e69928ce1d855ebb37cdb4e2c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, vendor=Red Hat, Inc., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T07:38:15.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:15.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: osdmap e121: 8 total, 7 up, 8 in 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[67016]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:38:15.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: osdmap e121: 8 total, 7 up, 8 in 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:15 smithi064 ceph-mon[64470]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: osdmap e121: 8 total, 7 up, 8 in 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:15 smithi136 ceph-mon[57888]: failed to encode map e121 with expected crc 2024-02-10T07:38:15.951 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77467]: 2024-02-10 07:38:15.572898062 +0000 UTC m=+0.018465695 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:38:15.951 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77467]: 2024-02-10 07:38:15.682338071 +0000 UTC m=+0.127905677 container create 9638489b5368fac12daa39e4b7ddb24ba87bd410c7fec9db929804fe3cb9b1f7 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:38:15.951 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77467]: 2024-02-10 07:38:15.876120253 +0000 UTC m=+0.321687860 container init 9638489b5368fac12daa39e4b7ddb24ba87bd410c7fec9db929804fe3cb9b1f7 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:38:15.951 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77467]: 2024-02-10 07:38:15.879480592 +0000 UTC m=+0.325048202 container start 9638489b5368fac12daa39e4b7ddb24ba87bd410c7fec9db929804fe3cb9b1f7 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T07:38:15.951 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:15 smithi064 podman[77467]: 2024-02-10 07:38:15.951099944 +0000 UTC m=+0.396667551 container attach 9638489b5368fac12daa39e4b7ddb24ba87bd410c7fec9db929804fe3cb9b1f7 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:38:16.205 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:16 smithi064 podman[77467]: 2024-02-10 07:38:16.177659707 +0000 UTC m=+0.623227307 container died 9638489b5368fac12daa39e4b7ddb24ba87bd410c7fec9db929804fe3cb9b1f7 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:38:16.521 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:16.521 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.521 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.521 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: osdmap e122: 8 total, 7 up, 8 in 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[67016]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: osdmap e122: 8 total, 7 up, 8 in 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.522 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.523 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:16 smithi064 ceph-mon[64470]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: osdmap e122: 8 total, 7 up, 8 in 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:16 smithi136 ceph-mon[57888]: failed to encode map e122 with expected crc 2024-02-10T07:38:16.782 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:16 smithi064 podman[77493]: 2024-02-10 07:38:16.763196685 +0000 UTC m=+0.580188048 container remove 9638489b5368fac12daa39e4b7ddb24ba87bd410c7fec9db929804fe3cb9b1f7 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131) 2024-02-10T07:38:17.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:16 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-10T07:38:17.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:16 smithi064 systemd[1]: Stopped Ceph osd.3 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:38:17.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:16 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.3.service: Consumed 8.533s CPU time. 2024-02-10T07:38:17.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:16 smithi064 systemd[1]: Starting Ceph osd.3 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:38:17.525 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:38:17] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.33.4" 2024-02-10T07:38:17.525 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-mon[67016]: pgmap v111: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:17.525 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 podman[77599]: 2024-02-10 07:38:17.116847499 +0000 UTC m=+0.018729919 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:38:17.526 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 podman[77599]: 2024-02-10 07:38:17.213924818 +0000 UTC m=+0.115807237 container create 0ce7ce8b4689610ba427771914d7ccb557439d2712ddbf639fadf12a503b1a60 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T07:38:17.526 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 podman[77599]: 2024-02-10 07:38:17.441622509 +0000 UTC m=+0.343504929 container init 0ce7ce8b4689610ba427771914d7ccb557439d2712ddbf639fadf12a503b1a60 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T07:38:17.526 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 podman[77599]: 2024-02-10 07:38:17.444705892 +0000 UTC m=+0.346588319 container start 0ce7ce8b4689610ba427771914d7ccb557439d2712ddbf639fadf12a503b1a60 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T07:38:17.526 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 podman[77599]: 2024-02-10 07:38:17.524657074 +0000 UTC m=+0.426539496 container attach 0ce7ce8b4689610ba427771914d7ccb557439d2712ddbf639fadf12a503b1a60 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T07:38:17.526 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-mon[64470]: pgmap v111: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:17.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:17 smithi136 ceph-mon[57888]: pgmap v111: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:18.194 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T07:38:18.195 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 bash[77599]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate[77614]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T07:38:18.196 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:17 smithi064 podman[77599]: 2024-02-10 07:38:17.913594664 +0000 UTC m=+0.815477074 container died 0ce7ce8b4689610ba427771914d7ccb557439d2712ddbf639fadf12a503b1a60 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:38:18.769 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 podman[77736]: 2024-02-10 07:38:18.488762546 +0000 UTC m=+0.570410914 container remove 0ce7ce8b4689610ba427771914d7ccb557439d2712ddbf639fadf12a503b1a60 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T07:38:18.769 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 podman[77804]: 2024-02-10 07:38:18.687512494 +0000 UTC m=+0.100271254 container create ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T07:38:18.769 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 podman[77804]: 2024-02-10 07:38:18.60489136 +0000 UTC m=+0.017650081 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:38:18.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:38:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:38:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:38:19.289 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 podman[77804]: 2024-02-10 07:38:18.865234901 +0000 UTC m=+0.277993630 container init ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-10T07:38:19.290 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 podman[77804]: 2024-02-10 07:38:18.868444431 +0000 UTC m=+0.281203152 container start ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:38:19.290 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 ceph-osd[77823]: -- 172.21.15.64:0/1356578343 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c253e731e0 con 0x55c253e44c00 2024-02-10T07:38:19.290 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 bash[77804]: ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 2024-02-10T07:38:19.290 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:18 smithi064 systemd[1]: Started Ceph osd.3 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:38:19.564 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:38:19.419+0000 7f0bd48fa7c0 -1 Falling back to public interface 2024-02-10T07:38:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-mon[64470]: pgmap v112: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:38:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-mon[67016]: pgmap v112: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:38:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:19.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:19.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:19 smithi136 ceph-mon[57888]: pgmap v112: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:38:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:20.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:20 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:20.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:20 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:20.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:20 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:21.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:21 smithi064 ceph-mon[64470]: pgmap v113: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:21.671 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:21 smithi064 ceph-mon[67016]: pgmap v113: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:21.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:21 smithi136 ceph-mon[57888]: pgmap v113: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:22.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-mon[64470]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:22.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-mon[64470]: from='osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:38:22.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-mon[64470]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:38:22.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-mon[67016]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:22.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-mon[67016]: from='osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:38:22.398 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-mon[67016]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:38:22.398 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:38:22.119+0000 7f0bd48fa7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-10T07:38:22.398 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:38:22.145+0000 7f0bd48fa7c0 -1 osd.3 120 log_to_monitors true 2024-02-10T07:38:22.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:22 smithi136 ceph-mon[57888]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:22 smithi136 ceph-mon[57888]: from='osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:38:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:22 smithi136 ceph-mon[57888]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:38:23.229 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:38:22 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:38:22.818+0000 7f0bc1d4a700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:38:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:38:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: failed to encode map e123 with expected crc 2024-02-10T07:38:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: osdmap e123: 8 total, 7 up, 8 in 2024-02-10T07:38:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: from='osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: from='osd.3 ' entity='osd.3' 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[64470]: failed to encode map e123 with expected crc 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: failed to encode map e123 with expected crc 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: osdmap e123: 8 total, 7 up, 8 in 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: from='osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: from='osd.3 ' entity='osd.3' 2024-02-10T07:38:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:23 smithi064 ceph-mon[67016]: failed to encode map e123 with expected crc 2024-02-10T07:38:23.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:38:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: failed to encode map e123 with expected crc 2024-02-10T07:38:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: osdmap e123: 8 total, 7 up, 8 in 2024-02-10T07:38:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: from='osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:38:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T07:38:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: from='osd.3 ' entity='osd.3' 2024-02-10T07:38:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:23 smithi136 ceph-mon[57888]: failed to encode map e123 with expected crc 2024-02-10T07:38:24.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: failed to encode map e123 with expected crc 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: pgmap v116: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096] boot 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: failed to encode map e124 with expected crc 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:38:24.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:24 smithi136 ceph-mon[57888]: failed to encode map e124 with expected crc 2024-02-10T07:38:24.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: failed to encode map e123 with expected crc 2024-02-10T07:38:24.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: pgmap v116: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:24.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:24.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:24.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096] boot 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: failed to encode map e124 with expected crc 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[67016]: failed to encode map e124 with expected crc 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: failed to encode map e123 with expected crc 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: pgmap v116: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:38:24.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: osd.3 [v2:172.21.15.64:6826/2011515096,v1:172.21.15.64:6827/2011515096] boot 2024-02-10T07:38:24.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T07:38:24.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: failed to encode map e124 with expected crc 2024-02-10T07:38:24.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:38:24.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:24 smithi064 ceph-mon[64470]: failed to encode map e124 with expected crc 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e124 with expected crc 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e124 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.677 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.678 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:25 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e124 with expected crc 2024-02-10T07:38:25.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:25.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:25 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.049 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:38:26.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[64470]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[64470]: pgmap v119: 161 pgs: 18 peering, 18 active+undersized+degraded, 30 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:38:26.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[64470]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:26.625 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.625 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.625 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[67016]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.625 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[67016]: pgmap v119: 161 pgs: 18 peering, 18 active+undersized+degraded, 30 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:38:26.625 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:26 smithi064 ceph-mon[67016]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:26 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:26 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:26 smithi136 ceph-mon[57888]: failed to encode map e125 with expected crc 2024-02-10T07:38:26.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:26 smithi136 ceph-mon[57888]: pgmap v119: 161 pgs: 18 peering, 18 active+undersized+degraded, 30 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:38:26.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:26 smithi136 ceph-mon[57888]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (4m) 3s ago 9m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (4m) 2m ago 9m 47.3M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (3m) 3s ago 8m 54.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (4m) 3s ago 16m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (3m) 3s ago 16m 49.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (2m) 2m ago 14m 20.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (2m) 3s ago 14m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (5m) 3s ago 10m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (4m) 2m ago 10m 17.3M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (113s) 3s ago 13m 44.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:38:26.738 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (77s) 3s ago 13m 44.0M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (42s) 3s ago 13m 40.7M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (8s) 3s ago 12m 12.3M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (12m) 2m ago 12m 53.1M 2992M 17.2.0 e1d6a67b021e 15ecfa4cba65 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (12m) 2m ago 12m 49.9M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (11m) 2m ago 11m 50.2M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (11m) 2m ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (3m) 2m ago 10m 54.8M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (8m) 3s ago 8m 74.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (8m) 2m ago 8m 72.4M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (8m) 3s ago 8m 73.1M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:38:26.739 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (8m) 2m ago 8m 71.9M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T07:38:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 9 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:38:27.152 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:38:27.501 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "9/23 daemons upgraded", 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:38:27.502 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:38:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:27 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:27.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:27 smithi136 ceph-mon[57888]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:27.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:27 smithi136 ceph-mon[57888]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:27.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:27 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1035068995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:27.810 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:38:27] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.33.4" 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[64470]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[64470]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1035068995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[67016]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[67016]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:27.811 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:27 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1035068995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: pg 6.c is active+undersized+degraded, acting [6,5] 2024-02-10T07:38:28.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:38:28.325+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:38:28.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: pgmap v120: 161 pgs: 18 peering, 18 active+undersized+degraded, 30 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:38:28.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:28.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2348008808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: pgmap v120: 161 pgs: 18 peering, 18 active+undersized+degraded, 30 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:38:28.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2348008808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:28.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:28 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:38:28.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: pgmap v120: 161 pgs: 18 peering, 18 active+undersized+degraded, 30 active+undersized, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:38:28.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2348008808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:28.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:38:28.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:38:28 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:38:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:38:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:29.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:29.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:29.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:29.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:29.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:30.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:30 smithi136 ceph-mon[57888]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:38:30.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:30 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded) 2024-02-10T07:38:30.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:30 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:38:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:30 smithi064 ceph-mon[64470]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:38:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:30 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded) 2024-02-10T07:38:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:30 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:38:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:30 smithi064 ceph-mon[67016]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:38:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:30 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded) 2024-02-10T07:38:30.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:30 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:38:32.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:32 smithi136 ceph-mon[57888]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:38:32.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:32 smithi064 ceph-mon[67016]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:38:32.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:32 smithi064 ceph-mon[64470]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:38:35.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:35 smithi064 ceph-mon[64470]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 320 B/s rd, 0 op/s 2024-02-10T07:38:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:35 smithi064 ceph-mon[67016]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 320 B/s rd, 0 op/s 2024-02-10T07:38:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:35 smithi136 ceph-mon[57888]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 320 B/s rd, 0 op/s 2024-02-10T07:38:36.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:36 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:36 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:36.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:36 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:37.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:37 smithi064 ceph-mon[64470]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 290 B/s rd, 0 op/s 2024-02-10T07:38:37.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:37 smithi064 ceph-mon[67016]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 290 B/s rd, 0 op/s 2024-02-10T07:38:37.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:38:37] "GET /metrics HTTP/1.1" 200 38517 "" "Prometheus/2.33.4" 2024-02-10T07:38:37.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:37 smithi136 ceph-mon[57888]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 290 B/s rd, 0 op/s 2024-02-10T07:38:38.972 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:38:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:38:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"smithi064:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.64\", device_class=\"ssd\", hostname=\"smithi064\", instance=\"172.21.15.136:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.64\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-10T07:38:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:39 smithi064 ceph-mon[64470]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:38:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:39 smithi064 ceph-mon[67016]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:38:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:39.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:39 smithi136 ceph-mon[57888]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:38:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:41 smithi064 ceph-mon[64470]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:38:41.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:41 smithi064 ceph-mon[67016]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:38:41.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:41 smithi136 ceph-mon[57888]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:38:43.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:43 smithi064 ceph-mon[64470]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:43.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:43 smithi064 ceph-mon[67016]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:43.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:43 smithi136 ceph-mon[57888]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[64470]: Upgrade: osd.4 is safe to restart 2024-02-10T07:38:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[67016]: Upgrade: osd.4 is safe to restart 2024-02-10T07:38:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:44 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T07:38:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:44 smithi136 ceph-mon[57888]: Upgrade: osd.4 is safe to restart 2024-02-10T07:38:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[64470]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:38:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:45.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[67016]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:45.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:45.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:38:45.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:45 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:45 smithi136 ceph-mon[57888]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:45 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:45 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:38:45.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:45 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:46.445 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:46 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:46.445 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:46 smithi136 ceph-mon[57888]: Upgrade: Updating osd.4 2024-02-10T07:38:46.446 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:46 smithi136 ceph-mon[57888]: Deploying daemon osd.4 on smithi136 2024-02-10T07:38:46.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:46 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:46.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:46 smithi064 ceph-mon[64470]: Upgrade: Updating osd.4 2024-02-10T07:38:46.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:46 smithi064 ceph-mon[64470]: Deploying daemon osd.4 on smithi136 2024-02-10T07:38:46.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:46 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:46.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:46 smithi064 ceph-mon[67016]: Upgrade: Updating osd.4 2024-02-10T07:38:46.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:46 smithi064 ceph-mon[67016]: Deploying daemon osd.4 on smithi136 2024-02-10T07:38:47.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:47 smithi136 ceph-mon[57888]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:47 smithi064 ceph-mon[64470]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:47.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:47 smithi064 ceph-mon[67016]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:47.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:38:47] "GET /metrics HTTP/1.1" 200 38515 "" "Prometheus/2.33.4" 2024-02-10T07:38:48.722 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:48 smithi136 systemd[1]: Stopping Ceph osd.4 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:38:49.222 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[28936]: 2024-02-10T07:38:48.757+0000 7f70352e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:38:49.223 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[28936]: 2024-02-10T07:38:48.757+0000 7f70352e0700 -1 osd.4 125 *** Got signal Terminated *** 2024-02-10T07:38:49.223 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:48 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[28936]: 2024-02-10T07:38:48.757+0000 7f70352e0700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:38:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:49 smithi064 ceph-mon[67016]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:49 smithi064 ceph-mon[67016]: osd.4 marked itself down and dead 2024-02-10T07:38:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:49 smithi064 ceph-mon[64470]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:49 smithi064 ceph-mon[64470]: osd.4 marked itself down and dead 2024-02-10T07:38:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:49 smithi136 ceph-mon[57888]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:49 smithi136 ceph-mon[57888]: osd.4 marked itself down and dead 2024-02-10T07:38:50.159 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:49 smithi136 podman[60684]: 2024-02-10 07:38:49.851410731 +0000 UTC m=+1.225497270 container died 15ecfa4cba65c01ace5d9ff4258a8ffcd3a966919f9854356c71c57d2c07bdbe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, vendor=Red Hat, Inc., version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T07:38:50.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:50 smithi136 podman[60684]: 2024-02-10 07:38:50.358589064 +0000 UTC m=+1.732675647 container cleanup 15ecfa4cba65c01ace5d9ff4258a8ffcd3a966919f9854356c71c57d2c07bdbe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, architecture=x86_64, RELEASE=HEAD, release=754, GIT_CLEAN=True, distribution-scope=public, maintainer=Guillaume Abrioux , name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_BRANCH=HEAD) 2024-02-10T07:38:50.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:50 smithi136 bash[60684]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4 2024-02-10T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:50 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:50 smithi136 ceph-mon[57888]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:50 smithi136 ceph-mon[57888]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:50 smithi136 ceph-mon[57888]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:50 smithi136 ceph-mon[57888]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:50 smithi136 ceph-mon[57888]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:38:50.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[64470]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[64470]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T07:38:50.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[64470]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[64470]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[64470]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[67016]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[67016]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[67016]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[67016]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:50 smithi064 ceph-mon[67016]: failed to encode map e126 with expected crc 2024-02-10T07:38:50.900 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:50 smithi136 podman[60702]: 2024-02-10 07:38:50.576602582 +0000 UTC m=+0.723434784 container remove 15ecfa4cba65c01ace5d9ff4258a8ffcd3a966919f9854356c71c57d2c07bdbe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, release=754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T07:38:51.223 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:50 smithi136 podman[60795]: 2024-02-10 07:38:50.803458844 +0000 UTC m=+0.018740460 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:38:51.223 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:50 smithi136 podman[60795]: 2024-02-10 07:38:50.927484896 +0000 UTC m=+0.142766510 container create bfb7001801e849ff18b7fd6f09c2910ddc8ff666154638330da7315af7cb20d1 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T07:38:51.223 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:51 smithi136 podman[60795]: 2024-02-10 07:38:51.146761722 +0000 UTC m=+0.362043336 container init bfb7001801e849ff18b7fd6f09c2910ddc8ff666154638330da7315af7cb20d1 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T07:38:51.223 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:51 smithi136 podman[60795]: 2024-02-10 07:38:51.150612888 +0000 UTC m=+0.365894533 container start bfb7001801e849ff18b7fd6f09c2910ddc8ff666154638330da7315af7cb20d1 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T07:38:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: pgmap v132: 161 pgs: 28 peering, 133 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:38:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T07:38:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[64470]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: pgmap v132: 161 pgs: 28 peering, 133 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:38:51.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:51 smithi064 ceph-mon[67016]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: pgmap v132: 161 pgs: 28 peering, 133 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:38:51.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T07:38:51.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T07:38:51.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.728 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.729 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.729 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:51 smithi136 ceph-mon[57888]: failed to encode map e127 with expected crc 2024-02-10T07:38:51.729 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:51 smithi136 podman[60795]: 2024-02-10 07:38:51.255841749 +0000 UTC m=+0.471123365 container attach bfb7001801e849ff18b7fd6f09c2910ddc8ff666154638330da7315af7cb20d1 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T07:38:51.729 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:51 smithi136 podman[60795]: 2024-02-10 07:38:51.441163463 +0000 UTC m=+0.656445069 container died bfb7001801e849ff18b7fd6f09c2910ddc8ff666154638330da7315af7cb20d1 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2) 2024-02-10T07:38:52.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 podman[60821]: 2024-02-10 07:38:52.150727508 +0000 UTC m=+0.702311090 container remove bfb7001801e849ff18b7fd6f09c2910ddc8ff666154638330da7315af7cb20d1 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T07:38:52.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-10T07:38:52.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 systemd[1]: Stopped Ceph osd.4 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:38:52.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.4.service: Consumed 8.555s CPU time. 2024-02-10T07:38:52.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 systemd[1]: Starting Ceph osd.4 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:38:52.920 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 podman[60926]: 2024-02-10 07:38:52.497538798 +0000 UTC m=+0.017666289 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:38:52.920 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 podman[60926]: 2024-02-10 07:38:52.601803307 +0000 UTC m=+0.121930808 container create 7f9553cfb3078a6d3c28b0d856979f542a6ec3b085d0ec868c403e1ee2d9e249 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T07:38:52.921 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 podman[60926]: 2024-02-10 07:38:52.813114917 +0000 UTC m=+0.333242419 container init 7f9553cfb3078a6d3c28b0d856979f542a6ec3b085d0ec868c403e1ee2d9e249 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T07:38:52.921 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 podman[60926]: 2024-02-10 07:38:52.816374503 +0000 UTC m=+0.336502000 container start 7f9553cfb3078a6d3c28b0d856979f542a6ec3b085d0ec868c403e1ee2d9e249 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a) 2024-02-10T07:38:52.921 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:52 smithi136 podman[60926]: 2024-02-10 07:38:52.92083846 +0000 UTC m=+0.440965976 container attach 7f9553cfb3078a6d3c28b0d856979f542a6ec3b085d0ec868c403e1ee2d9e249 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T07:38:53.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:53 smithi064 ceph-mon[64470]: pgmap v134: 161 pgs: 19 stale+active+clean, 28 peering, 114 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:53.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:53 smithi064 ceph-mon[67016]: pgmap v134: 161 pgs: 19 stale+active+clean, 28 peering, 114 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:53.574 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-mon[57888]: pgmap v134: 161 pgs: 19 stale+active+clean, 28 peering, 114 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 bash[60926]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T07:38:53.575 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T07:38:53.576 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T07:38:53.576 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate[60941]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T07:38:53.576 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 podman[60926]: 2024-02-10 07:38:53.283450416 +0000 UTC m=+0.803577920 container died 7f9553cfb3078a6d3c28b0d856979f542a6ec3b085d0ec868c403e1ee2d9e249 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T07:38:54.196 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:53 smithi136 podman[61074]: 2024-02-10 07:38:53.874424202 +0000 UTC m=+0.585169337 container remove 7f9553cfb3078a6d3c28b0d856979f542a6ec3b085d0ec868c403e1ee2d9e249 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T07:38:54.196 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 podman[61140]: 2024-02-10 07:38:53.987509189 +0000 UTC m=+0.017475577 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:38:54.197 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 podman[61140]: 2024-02-10 07:38:54.100267787 +0000 UTC m=+0.130234200 container create 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-10T07:38:54.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:54 smithi136 ceph-mon[57888]: pgmap v135: 161 pgs: 11 stale+active+clean, 8 active+undersized+degraded, 28 peering, 11 active+undersized, 103 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 30/627 objects degraded (4.785%) 2024-02-10T07:38:54.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:54 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 30/627 objects degraded (4.785%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:54.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 podman[61140]: 2024-02-10 07:38:54.294020624 +0000 UTC m=+0.323987011 container init 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:38:54.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 podman[61140]: 2024-02-10 07:38:54.297047399 +0000 UTC m=+0.327013783 container start 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a) 2024-02-10T07:38:54.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 ceph-osd[61159]: -- 172.21.15.136:0/3705261916 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f34c171e0 con 0x559f34be9000 2024-02-10T07:38:54.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 bash[61140]: 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a 2024-02-10T07:38:54.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 systemd[1]: Started Ceph osd.4 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:38:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:54 smithi064 ceph-mon[64470]: pgmap v135: 161 pgs: 11 stale+active+clean, 8 active+undersized+degraded, 28 peering, 11 active+undersized, 103 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 30/627 objects degraded (4.785%) 2024-02-10T07:38:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:54 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 30/627 objects degraded (4.785%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:54.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:54 smithi064 ceph-mon[67016]: pgmap v135: 161 pgs: 11 stale+active+clean, 8 active+undersized+degraded, 28 peering, 11 active+undersized, 103 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 30/627 objects degraded (4.785%) 2024-02-10T07:38:54.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:54 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 30/627 objects degraded (4.785%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:38:55.222 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:38:54.833+0000 7fa4375be7c0 -1 Falling back to public interface 2024-02-10T07:38:55.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:55.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:55.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:55 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:55 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:55.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:55 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:56 smithi136 ceph-mon[57888]: pgmap v136: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:38:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:56 smithi136 ceph-mon[57888]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T07:38:56.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:56 smithi064 ceph-mon[64470]: pgmap v136: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:38:56.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:56 smithi064 ceph-mon[64470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T07:38:56.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:56 smithi064 ceph-mon[67016]: pgmap v136: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:38:56.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:56 smithi064 ceph-mon[67016]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T07:38:57.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:38:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:38:57] "GET /metrics HTTP/1.1" 200 38515 "" "Prometheus/2.33.4" 2024-02-10T07:38:57.972 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:38:57.555+0000 7fa4375be7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-10T07:38:57.973 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:38:57.584+0000 7fa4375be7c0 -1 osd.4 125 log_to_monitors true 2024-02-10T07:38:58.290 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:38:58.525 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:38:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:38:58.469+0000 7fa428a16700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:38:58.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:58 smithi136 ceph-mon[57888]: pgmap v137: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:38:58.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:58 smithi136 ceph-mon[57888]: from='osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:38:58.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:58 smithi136 ceph-mon[57888]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:38:58.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:58 smithi136 ceph-mon[57888]: from='client.44391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:58.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[67016]: pgmap v137: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:38:58.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[67016]: from='osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:38:58.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[67016]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:38:58.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[67016]: from='client.44391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:58.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[64470]: pgmap v137: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:38:58.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[64470]: from='osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:38:58.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[64470]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:38:58.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:58 smithi064 ceph-mon[64470]: from='client.44391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (5m) 35s ago 10m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (4m) 0s ago 10m 55.0M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (4m) 35s ago 9m 54.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (4m) 0s ago 14m 450M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (4m) 35s ago 16m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:38:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (3m) 35s ago 16m 49.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (3m) 0s ago 15m 39.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (3m) 35s ago 15m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (5m) 35s ago 11m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (5m) 0s ago 10m 17.1M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (2m) 35s ago 14m 44.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (109s) 35s ago 14m 44.0M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (75s) 35s ago 13m 40.7M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (40s) 35s ago 13m 12.3M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (4s) 0s ago 12m 15.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (12m) 0s ago 12m 56.5M 2992M 17.2.0 e1d6a67b021e a65a818414a2 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (12m) 0s ago 12m 56.4M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (11m) 0s ago 11m 61.6M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (4m) 0s ago 11m 68.1M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:38:58.975 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (9m) 35s ago 9m 74.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:38:58.976 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (9m) 0s ago 9m 73.2M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:38:58.976 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (9m) 35s ago 9m 73.1M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:38:58.976 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (9m) 0s ago 9m 73.1M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:38:59.381 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:38:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:38:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 9 2024-02-10T07:38:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:38:59.383 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:38:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: failed to encode map e128 with expected crc 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: osdmap e128: 8 total, 7 up, 8 in 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: failed to encode map e128 with expected crc 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='client.54352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='client.44403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:38:59 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/881601163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:59.736 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:38:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:38:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:38:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:38:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "10/23 daemons upgraded", 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:38:59.737 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:38:59.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:59.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:38:59.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:38:59.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: failed to encode map e128 with expected crc 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: osdmap e128: 8 total, 7 up, 8 in 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: failed to encode map e128 with expected crc 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='client.54352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='client.44403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/881601163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:38:59.738 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: failed to encode map e128 with expected crc 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: osdmap e128: 8 total, 7 up, 8 in 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: failed to encode map e128 with expected crc 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='client.54352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='client.44403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:38:59.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:38:59 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/881601163' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN Degraded data redundancy: 107/627 objects degraded (17.065%), 28 pgs degraded 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 107/627 objects degraded (17.065%), 28 pgs degraded 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-02-10T07:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-02-10T07:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-02-10T07:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-02-10T07:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-02-10T07:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.e is active+undersized+degraded, acting [6,1] 2024-02-10T07:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: pg 5.e is active+undersized+degraded, acting [5,0] 2024-02-10T07:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-02-10T07:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e128 with expected crc 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: pgmap v139: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525] boot 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:00.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:00.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:00.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:00 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/3524299994' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:39:00.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e128 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: pgmap v139: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525] boot 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:00.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3524299994' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e128 with expected crc 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: pgmap v139: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: osd.4 [v2:172.21.15.136:6800/3413103525,v1:172.21.15.136:6801/3413103525] boot 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: failed to encode map e129 with expected crc 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:00.817 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:00 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/3524299994' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.664 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.664 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:01 smithi136 ceph-mon[57888]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T07:39:01.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[67016]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:01.816 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:01 smithi064 ceph-mon[64470]: failed to encode map e130 with expected crc 2024-02-10T07:39:02.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:02 smithi136 ceph-mon[57888]: pgmap v142: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 107/627 objects degraded (17.065%) 2024-02-10T07:39:02.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:02 smithi064 ceph-mon[64470]: pgmap v142: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 107/627 objects degraded (17.065%) 2024-02-10T07:39:02.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:02 smithi064 ceph-mon[67016]: pgmap v142: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 107/627 objects degraded (17.065%) 2024-02-10T07:39:03.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:39:03.311+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:04.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:04 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:39:04.543 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:04.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:04.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:04.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:04.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:04.546 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:04 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-10T07:39:05.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:05 smithi136 ceph-mon[57888]: pgmap v143: 161 pgs: 23 active+undersized+degraded, 35 active+undersized, 103 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 87/627 objects degraded (13.876%) 2024-02-10T07:39:05.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:05 smithi064 ceph-mon[64470]: pgmap v143: 161 pgs: 23 active+undersized+degraded, 35 active+undersized, 103 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 87/627 objects degraded (13.876%) 2024-02-10T07:39:05.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:05 smithi064 ceph-mon[67016]: pgmap v143: 161 pgs: 23 active+undersized+degraded, 35 active+undersized, 103 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 87/627 objects degraded (13.876%) 2024-02-10T07:39:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:06 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:06.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:06 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:06.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:06 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-mon[64470]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-10T07:39:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/627 objects degraded (13.876%), 23 pgs degraded) 2024-02-10T07:39:07.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:39:07.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:39:07] "GET /metrics HTTP/1.1" 200 38515 "" "Prometheus/2.33.4" 2024-02-10T07:39:07.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-mon[67016]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-10T07:39:07.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/627 objects degraded (13.876%), 23 pgs degraded) 2024-02-10T07:39:07.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:07 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:39:07.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:07 smithi136 ceph-mon[57888]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-10T07:39:07.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:07 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/627 objects degraded (13.876%), 23 pgs degraded) 2024-02-10T07:39:07.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:07 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:39:09.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:09 smithi064 ceph-mon[64470]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:39:09.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:09 smithi064 ceph-mon[67016]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:39:09.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:09 smithi136 ceph-mon[57888]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:39:11.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:11 smithi064 ceph-mon[64470]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:39:11.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:11 smithi064 ceph-mon[67016]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:39:11.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:11 smithi136 ceph-mon[57888]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:39:13.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:13 smithi064 ceph-mon[64470]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-10T07:39:13.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:13 smithi064 ceph-mon[67016]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-10T07:39:13.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:13 smithi136 ceph-mon[57888]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-10T07:39:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:14 smithi136 ceph-mon[57888]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[64470]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:14.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:14.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[67016]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:14.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:15.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:15 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:15.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:15 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:15.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:15 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:16.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:16 smithi136 ceph-mon[57888]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:16.733 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:16 smithi064 ceph-mon[64470]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:16.734 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:16 smithi064 ceph-mon[67016]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:17.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:39:17] "GET /metrics HTTP/1.1" 200 38521 "" "Prometheus/2.33.4" 2024-02-10T07:39:19.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:19 smithi136 ceph-mon[57888]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:19 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:19 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:19.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:19 smithi136 ceph-mon[57888]: Upgrade: osd.5 is safe to restart 2024-02-10T07:39:19.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[67016]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:19.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:19.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:19.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[67016]: Upgrade: osd.5 is safe to restart 2024-02-10T07:39:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[64470]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T07:39:19.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:19 smithi064 ceph-mon[64470]: Upgrade: osd.5 is safe to restart 2024-02-10T07:39:21.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:20 smithi136 ceph-mon[57888]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:21.151 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:20 smithi136 ceph-mon[57888]: Upgrade: Updating osd.5 2024-02-10T07:39:21.151 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:21.151 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:39:21.151 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:20 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:21.151 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:20 smithi136 ceph-mon[57888]: Deploying daemon osd.5 on smithi136 2024-02-10T07:39:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[64470]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[64470]: Upgrade: Updating osd.5 2024-02-10T07:39:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[64470]: Deploying daemon osd.5 on smithi136 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[67016]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[67016]: Upgrade: Updating osd.5 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:21.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:20 smithi064 ceph-mon[67016]: Deploying daemon osd.5 on smithi136 2024-02-10T07:39:23.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:22 smithi136 ceph-mon[57888]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:23.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:22 smithi064 ceph-mon[64470]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:23.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:22 smithi064 ceph-mon[67016]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:23.529 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:23 smithi136 systemd[1]: Stopping Ceph osd.5 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:39:23.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[31871]: 2024-02-10T07:39:23.528+0000 7f3a11312700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:39:23.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[31871]: 2024-02-10T07:39:23.528+0000 7f3a11312700 -1 osd.5 130 *** Got signal Terminated *** 2024-02-10T07:39:23.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[31871]: 2024-02-10T07:39:23.528+0000 7f3a11312700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:39:24.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:24 smithi064 ceph-mon[64470]: osd.5 marked itself down and dead 2024-02-10T07:39:24.553 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:24 smithi064 ceph-mon[67016]: osd.5 marked itself down and dead 2024-02-10T07:39:24.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:24 smithi136 ceph-mon[57888]: osd.5 marked itself down and dead 2024-02-10T07:39:24.904 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:24 smithi136 podman[63023]: 2024-02-10 07:39:24.614084462 +0000 UTC m=+1.224316884 container died a65a818414a27c0cbcc8f50a399d9fe015b6e41301fab703299bacb28f270139 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, ceph=True, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T07:39:25.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:24 smithi136 podman[63023]: 2024-02-10 07:39:24.986736273 +0000 UTC m=+1.596968701 container cleanup a65a818414a27c0cbcc8f50a399d9fe015b6e41301fab703299bacb28f270139 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, architecture=x86_64, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, GIT_BRANCH=HEAD, distribution-scope=public) 2024-02-10T07:39:25.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:24 smithi136 bash[63023]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5 2024-02-10T07:39:25.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:25.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:39:25.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: osdmap e131: 8 total, 7 up, 8 in 2024-02-10T07:39:25.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[67016]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: osdmap e131: 8 total, 7 up, 8 in 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:25 smithi064 ceph-mon[64470]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.566 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:25 smithi136 podman[63041]: 2024-02-10 07:39:25.237634453 +0000 UTC m=+0.625257294 container remove a65a818414a27c0cbcc8f50a399d9fe015b6e41301fab703299bacb28f270139 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, release=754, version=8, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: osdmap e131: 8 total, 7 up, 8 in 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.566 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:25 smithi136 ceph-mon[57888]: failed to encode map e131 with expected crc 2024-02-10T07:39:25.841 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:25 smithi136 podman[63133]: 2024-02-10 07:39:25.468560097 +0000 UTC m=+0.019278534 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:39:25.841 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:25 smithi136 podman[63133]: 2024-02-10 07:39:25.630223062 +0000 UTC m=+0.180941499 container create 92fff2efc5a57fb81016d693f9fa621ad4e1a102059ad5bbd58bb7598d40a400 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T07:39:25.841 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:25 smithi136 podman[63133]: 2024-02-10 07:39:25.841175666 +0000 UTC m=+0.391894106 container init 92fff2efc5a57fb81016d693f9fa621ad4e1a102059ad5bbd58bb7598d40a400 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T07:39:26.135 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:25 smithi136 podman[63133]: 2024-02-10 07:39:25.844646861 +0000 UTC m=+0.395365302 container start 92fff2efc5a57fb81016d693f9fa621ad4e1a102059ad5bbd58bb7598d40a400 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a) 2024-02-10T07:39:26.136 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:25 smithi136 podman[63133]: 2024-02-10 07:39:25.941207203 +0000 UTC m=+0.491925658 container attach 92fff2efc5a57fb81016d693f9fa621ad4e1a102059ad5bbd58bb7598d40a400 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2) 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: osdmap e132: 8 total, 7 up, 8 in 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:26 smithi136 ceph-mon[57888]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.474 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:26 smithi136 podman[63133]: 2024-02-10 07:39:26.136934295 +0000 UTC m=+0.687652743 container died 92fff2efc5a57fb81016d693f9fa621ad4e1a102059ad5bbd58bb7598d40a400 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-10T07:39:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: osdmap e132: 8 total, 7 up, 8 in 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[64470]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: osdmap e132: 8 total, 7 up, 8 in 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: failed to encode map e132 with expected crc 2024-02-10T07:39:26.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:26 smithi064 ceph-mon[67016]: failed to encode map e132 with expected crc 2024-02-10T07:39:27.199 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:26 smithi136 podman[63158]: 2024-02-10 07:39:26.76121557 +0000 UTC m=+0.615039613 container remove 92fff2efc5a57fb81016d693f9fa621ad4e1a102059ad5bbd58bb7598d40a400 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:39:27.199 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:26 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-10T07:39:27.199 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:26 smithi136 systemd[1]: Stopped Ceph osd.5 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:39:27.200 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:26 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.5.service: Consumed 8.282s CPU time. 2024-02-10T07:39:27.200 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:26 smithi136 systemd[1]: Starting Ceph osd.5 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:39:27.451 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-mon[57888]: pgmap v156: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:27.451 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-mon[57888]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T07:39:27.452 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 podman[63262]: 2024-02-10 07:39:27.102515216 +0000 UTC m=+0.017944039 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:39:27.452 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 podman[63262]: 2024-02-10 07:39:27.217410194 +0000 UTC m=+0.132839043 container create 11774113fad29e31e6952456be35a3364a6a8ecf23d3abf5c110f9450adca87f (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T07:39:27.452 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 podman[63262]: 2024-02-10 07:39:27.448489246 +0000 UTC m=+0.363918064 container init 11774113fad29e31e6952456be35a3364a6a8ecf23d3abf5c110f9450adca87f (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T07:39:27.452 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 podman[63262]: 2024-02-10 07:39:27.451852233 +0000 UTC m=+0.367281042 container start 11774113fad29e31e6952456be35a3364a6a8ecf23d3abf5c110f9450adca87f (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T07:39:27.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:27 smithi064 ceph-mon[64470]: pgmap v156: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:27.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:27 smithi064 ceph-mon[64470]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T07:39:27.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:39:27] "GET /metrics HTTP/1.1" 200 38521 "" "Prometheus/2.33.4" 2024-02-10T07:39:27.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:27 smithi064 ceph-mon[67016]: pgmap v156: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:27.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:27 smithi064 ceph-mon[67016]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T07:39:27.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 podman[63262]: 2024-02-10 07:39:27.548340526 +0000 UTC m=+0.463769334 container attach 11774113fad29e31e6952456be35a3364a6a8ecf23d3abf5c110f9450adca87f (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T07:39:28.221 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:39:28.221 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 bash[63262]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate[63278]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T07:39:28.222 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:27 smithi136 podman[63262]: 2024-02-10 07:39:27.917332019 +0000 UTC m=+0.832760827 container died 11774113fad29e31e6952456be35a3364a6a8ecf23d3abf5c110f9450adca87f (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T07:39:28.786 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:28 smithi136 podman[63401]: 2024-02-10 07:39:28.526759493 +0000 UTC m=+0.604480484 container remove 11774113fad29e31e6952456be35a3364a6a8ecf23d3abf5c110f9450adca87f (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-activate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T07:39:28.786 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:28 smithi136 podman[63461]: 2024-02-10 07:39:28.644287074 +0000 UTC m=+0.018603394 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:39:28.786 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:28 smithi136 podman[63461]: 2024-02-10 07:39:28.786203406 +0000 UTC m=+0.160519760 container create f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2) 2024-02-10T07:39:29.080 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:28 smithi136 podman[63461]: 2024-02-10 07:39:28.97994738 +0000 UTC m=+0.354263693 container init f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T07:39:29.080 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:28 smithi136 podman[63461]: 2024-02-10 07:39:28.983684042 +0000 UTC m=+0.358000362 container start f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T07:39:29.080 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-osd[63479]: -- 172.21.15.136:0/1539038302 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564dc387d1e0 con 0x564dc384ec00 2024-02-10T07:39:29.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-mon[57888]: pgmap v157: 161 pgs: 5 active+undersized+degraded, 11 peering, 5 active+undersized, 140 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 19/627 objects degraded (3.030%) 2024-02-10T07:39:29.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 19/627 objects degraded (3.030%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:29.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.473 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:29 smithi136 bash[63461]: f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa 2024-02-10T07:39:29.473 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:29 smithi136 systemd[1]: Started Ceph osd.5 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:39:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[67016]: pgmap v157: 161 pgs: 5 active+undersized+degraded, 11 peering, 5 active+undersized, 140 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 19/627 objects degraded (3.030%) 2024-02-10T07:39:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 19/627 objects degraded (3.030%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[64470]: pgmap v157: 161 pgs: 5 active+undersized+degraded, 11 peering, 5 active+undersized, 140 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 19/627 objects degraded (3.030%) 2024-02-10T07:39:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 19/627 objects degraded (3.030%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:29.968 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:29 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:39:29.523+0000 7f92c7c647c0 -1 Falling back to public interface 2024-02-10T07:39:30.499 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:39:30.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:30 smithi136 ceph-mon[57888]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-10T07:39:30.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:30 smithi064 ceph-mon[64470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-10T07:39:30.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:30 smithi064 ceph-mon[67016]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-10T07:39:31.163 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (5m) 67s ago 10m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (5m) 32s ago 10m 55.0M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (4m) 67s ago 9m 54.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (4m) 32s ago 15m 450M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (5m) 67s ago 17m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (4m) 67s ago 17m 49.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (3m) 32s ago 15m 39.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (4m) 67s ago 15m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (6m) 67s ago 11m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (5m) 32s ago 11m 17.1M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (2m) 67s ago 15m 44.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (2m) 67s ago 14m 44.0M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:39:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (107s) 67s ago 14m 40.7M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (72s) 67s ago 13m 12.3M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (37s) 32s ago 13m 15.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 starting - - - 2992M 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (12m) 32s ago 12m 56.4M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (12m) 32s ago 12m 61.6M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (4m) 32s ago 11m 68.1M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (10m) 67s ago 10m 74.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (10m) 32s ago 9m 73.2M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (9m) 67s ago 9m 73.1M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:39:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (9m) 32s ago 9m 73.1M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:39:31.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:31 smithi064 ceph-mon[64470]: pgmap v158: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:31.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:31 smithi064 ceph-mon[67016]: pgmap v158: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T07:39:31.571 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 10 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:39:31.572 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:39:31.639 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:31 smithi136 ceph-mon[57888]: pgmap v158: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:31.943 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "10/23 daemons upgraded", 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:39:31.944 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:39:32.350 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 72/627 objects degraded (11.483%), 20 pgs degraded 2024-02-10T07:39:32.350 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: osd.5 (root=default,host=smithi136) is down 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 72/627 objects degraded (11.483%), 20 pgs degraded 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.1f is active+undersized+degraded, acting [6,1] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 5.e is active+undersized+degraded, acting [4,0] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-02-10T07:39:32.352 INFO:teuthology.orchestra.run.smithi064.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-02-10T07:39:32.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-mon[57888]: from='client.54382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-mon[57888]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-mon[57888]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2206661648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:32.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-mon[57888]: from='osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:39:32.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-mon[57888]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:39:32.617 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:39:32.224+0000 7f92c7c647c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-10T07:39:32.617 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:39:32.251+0000 7f92c7c647c0 -1 osd.5 130 log_to_monitors true 2024-02-10T07:39:32.617 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:39:32 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:39:32.616+0000 7f92b50b4700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:39:32.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[64470]: from='client.54382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[64470]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[64470]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2206661648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[64470]: from='osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[64470]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[67016]: from='client.54382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[67016]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[67016]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2206661648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[67016]: from='osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:39:32.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:32 smithi064 ceph-mon[67016]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:39:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 10 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: from='client.44454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: failed to encode map e133 with expected crc 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: failed to encode map e133 with expected crc 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: from='osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: osdmap e133: 8 total, 7 up, 8 in 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1095768119' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:39:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:33 smithi136 ceph-mon[57888]: from='osd.5 ' entity='osd.5' 2024-02-10T07:39:33.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 10 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:33.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: from='client.44454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:33.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:39:33.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: failed to encode map e133 with expected crc 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: failed to encode map e133 with expected crc 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: from='osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: osdmap e133: 8 total, 7 up, 8 in 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1095768119' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[64470]: from='osd.5 ' entity='osd.5' 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 10 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: from='client.44454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: failed to encode map e133 with expected crc 2024-02-10T07:39:33.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: failed to encode map e133 with expected crc 2024-02-10T07:39:33.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: from='osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:39:33.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: osdmap e133: 8 total, 7 up, 8 in 2024-02-10T07:39:33.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:39:33.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1095768119' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:39:33.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:33 smithi064 ceph-mon[67016]: from='osd.5 ' entity='osd.5' 2024-02-10T07:39:34.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 83 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:34.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:34.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:34.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:39:34.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760] boot 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[64470]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 83 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760] boot 2024-02-10T07:39:34.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T07:39:34.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:39:34.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:34 smithi064 ceph-mon[67016]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 83 op/s; 72/627 objects degraded (11.483%) 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: osd.5 [v2:172.21.15.136:6808/215571760,v1:172.21.15.136:6809/215571760] boot 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: failed to encode map e134 with expected crc 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:39:34.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:34 smithi136 ceph-mon[57888]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:35.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:35.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:35.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: failed to encode map e135 with expected crc 2024-02-10T07:39:35.527 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T07:39:35.527 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:35 smithi136 ceph-mon[57888]: failed to encode map e135 with expected crc 2024-02-10T07:39:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: failed to encode map e135 with expected crc 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[64470]: failed to encode map e135 with expected crc 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: failed to encode map e134 with expected crc 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: failed to encode map e135 with expected crc 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T07:39:35.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:35 smithi064 ceph-mon[67016]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:36 smithi136 ceph-mon[57888]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:36 smithi136 ceph-mon[57888]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:36 smithi136 ceph-mon[57888]: pgmap v164: 161 pgs: 16 peering, 15 active+undersized+degraded, 28 active+undersized, 102 active+clean; 585 KiB data, 212 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 0 B/s wr, 219 op/s; 49/627 objects degraded (7.815%) 2024-02-10T07:39:36.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:36 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 49/627 objects degraded (7.815%), 15 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:36.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[64470]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[64470]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[64470]: pgmap v164: 161 pgs: 16 peering, 15 active+undersized+degraded, 28 active+undersized, 102 active+clean; 585 KiB data, 212 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 0 B/s wr, 219 op/s; 49/627 objects degraded (7.815%) 2024-02-10T07:39:36.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 49/627 objects degraded (7.815%), 15 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:36.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[67016]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[67016]: failed to encode map e135 with expected crc 2024-02-10T07:39:36.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[67016]: pgmap v164: 161 pgs: 16 peering, 15 active+undersized+degraded, 28 active+undersized, 102 active+clean; 585 KiB data, 212 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 0 B/s wr, 219 op/s; 49/627 objects degraded (7.815%) 2024-02-10T07:39:36.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:36 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 49/627 objects degraded (7.815%), 15 pgs degraded (PG_DEGRADED) 2024-02-10T07:39:37.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:39:37] "GET /metrics HTTP/1.1" 200 38522 "" "Prometheus/2.33.4" 2024-02-10T07:39:38.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:38 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:39:38.418+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-10T07:39:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: pgmap v165: 161 pgs: 16 peering, 5 active+undersized+degraded, 12 active+undersized, 128 active+clean; 585 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 KiB/s rd, 0 B/s wr, 283 op/s; 11/627 objects degraded (1.754%); 0 B/s, 4 objects/s recovering 2024-02-10T07:39:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: pgmap v165: 161 pgs: 16 peering, 5 active+undersized+degraded, 12 active+undersized, 128 active+clean; 585 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 KiB/s rd, 0 B/s wr, 283 op/s; 11/627 objects degraded (1.754%); 0 B/s, 4 objects/s recovering 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:39.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:39.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:39.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:39 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: pgmap v165: 161 pgs: 16 peering, 5 active+undersized+degraded, 12 active+undersized, 128 active+clean; 585 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 KiB/s rd, 0 B/s wr, 283 op/s; 11/627 objects degraded (1.754%); 0 B/s, 4 objects/s recovering 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:39:39.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:39 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:40.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:40 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:40.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:40 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-10T07:39:40.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:40 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:40.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:40 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-10T07:39:40.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:40 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:40.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:40 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-10T07:39:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:41 smithi064 ceph-mon[64470]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 225 KiB/s rd, 0 B/s wr, 374 op/s; 0 B/s, 7 objects/s recovering 2024-02-10T07:39:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:41 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/627 objects degraded (1.754%), 5 pgs degraded) 2024-02-10T07:39:41.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:41 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:39:41.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:41 smithi064 ceph-mon[67016]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 225 KiB/s rd, 0 B/s wr, 374 op/s; 0 B/s, 7 objects/s recovering 2024-02-10T07:39:41.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:41 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/627 objects degraded (1.754%), 5 pgs degraded) 2024-02-10T07:39:41.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:41 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:39:41.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:41 smithi136 ceph-mon[57888]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 225 KiB/s rd, 0 B/s wr, 374 op/s; 0 B/s, 7 objects/s recovering 2024-02-10T07:39:41.806 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:41 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/627 objects degraded (1.754%), 5 pgs degraded) 2024-02-10T07:39:41.806 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:41 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:39:43.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:43 smithi064 ceph-mon[64470]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 155 KiB/s rd, 0 B/s wr, 258 op/s; 0 B/s, 6 objects/s recovering 2024-02-10T07:39:43.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:43 smithi064 ceph-mon[67016]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 155 KiB/s rd, 0 B/s wr, 258 op/s; 0 B/s, 6 objects/s recovering 2024-02-10T07:39:43.730 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:43 smithi136 ceph-mon[57888]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 155 KiB/s rd, 0 B/s wr, 258 op/s; 0 B/s, 6 objects/s recovering 2024-02-10T07:39:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:44 smithi136 ceph-mon[57888]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 195 op/s; 0 B/s, 5 objects/s recovering 2024-02-10T07:39:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:44.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:44 smithi064 ceph-mon[64470]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 195 op/s; 0 B/s, 5 objects/s recovering 2024-02-10T07:39:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:44 smithi064 ceph-mon[67016]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 195 op/s; 0 B/s, 5 objects/s recovering 2024-02-10T07:39:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:45.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:45 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:45.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:45 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:45 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:46.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:46 smithi136 ceph-mon[57888]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s; 0 B/s, 4 objects/s recovering 2024-02-10T07:39:46.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:46 smithi064 ceph-mon[64470]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s; 0 B/s, 4 objects/s recovering 2024-02-10T07:39:46.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:46 smithi064 ceph-mon[67016]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s; 0 B/s, 4 objects/s recovering 2024-02-10T07:39:47.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:39:47] "GET /metrics HTTP/1.1" 200 38529 "" "Prometheus/2.33.4" 2024-02-10T07:39:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:49 smithi064 ceph-mon[64470]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s; 0 B/s, 4 objects/s recovering 2024-02-10T07:39:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:49.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:49 smithi064 ceph-mon[67016]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s; 0 B/s, 4 objects/s recovering 2024-02-10T07:39:49.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:49 smithi136 ceph-mon[57888]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s; 0 B/s, 4 objects/s recovering 2024-02-10T07:39:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:51.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:51 smithi064 ceph-mon[64470]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T07:39:51.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:51 smithi064 ceph-mon[67016]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T07:39:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:51 smithi136 ceph-mon[57888]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T07:39:53.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:53 smithi064 ceph-mon[64470]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:39:53.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:53 smithi064 ceph-mon[67016]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:39:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:53 smithi136 ceph-mon[57888]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:39:54.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:54 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:54.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:54 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:54.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:54 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[67016]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:39:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[67016]: Upgrade: osd.6 is safe to restart 2024-02-10T07:39:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:39:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[64470]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:39:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[64470]: Upgrade: osd.6 is safe to restart 2024-02-10T07:39:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:39:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:55.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:55 smithi136 ceph-mon[57888]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T07:39:55.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:55 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T07:39:55.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:55 smithi136 ceph-mon[57888]: Upgrade: osd.6 is safe to restart 2024-02-10T07:39:55.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:39:55.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:39:55.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:56 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:56 smithi064 ceph-mon[64470]: Upgrade: Updating osd.6 2024-02-10T07:39:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:56 smithi064 ceph-mon[64470]: Deploying daemon osd.6 on smithi136 2024-02-10T07:39:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:56 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:56 smithi064 ceph-mon[67016]: Upgrade: Updating osd.6 2024-02-10T07:39:56.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:56 smithi064 ceph-mon[67016]: Deploying daemon osd.6 on smithi136 2024-02-10T07:39:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:56 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:56 smithi136 ceph-mon[57888]: Upgrade: Updating osd.6 2024-02-10T07:39:56.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:56 smithi136 ceph-mon[57888]: Deploying daemon osd.6 on smithi136 2024-02-10T07:39:57.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:57 smithi136 ceph-mon[57888]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:57.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:57 smithi064 ceph-mon[64470]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:57.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:57 smithi064 ceph-mon[67016]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:39:57.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:39:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:39:57] "GET /metrics HTTP/1.1" 200 38529 "" "Prometheus/2.33.4" 2024-02-10T07:39:58.688 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:39:58 smithi136 systemd[1]: Stopping Ceph osd.6 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:39:58.972 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:39:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[34763]: 2024-02-10T07:39:58.688+0000 7f5048e20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:39:58.973 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:39:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[34763]: 2024-02-10T07:39:58.688+0000 7f5048e20700 -1 osd.6 135 *** Got signal Terminated *** 2024-02-10T07:39:58.973 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:39:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[34763]: 2024-02-10T07:39:58.688+0000 7f5048e20700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:39:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:59 smithi064 ceph-mon[64470]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:39:59 smithi064 ceph-mon[64470]: osd.6 marked itself down and dead 2024-02-10T07:39:59.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:59 smithi064 ceph-mon[67016]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:39:59 smithi064 ceph-mon[67016]: osd.6 marked itself down and dead 2024-02-10T07:39:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:59 smithi136 ceph-mon[57888]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:39:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:39:59 smithi136 ceph-mon[57888]: osd.6 marked itself down and dead 2024-02-10T07:40:00.064 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:39:59 smithi136 podman[65353]: 2024-02-10 07:39:59.781801197 +0000 UTC m=+1.222034285 container died 03001fcbe817e9a6fd93aa373ee7b00ee6369ef4f22ce7c49038056a7ff10f7c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vcs-type=git, vendor=Red Hat, Inc., RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.component=centos-stream-container, name=centos-stream) 2024-02-10T07:40:00.456 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:00 smithi136 podman[65353]: 2024-02-10 07:40:00.163434983 +0000 UTC m=+1.603668064 container cleanup 03001fcbe817e9a6fd93aa373ee7b00ee6369ef4f22ce7c49038056a7ff10f7c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.component=centos-stream-container, architecture=x86_64, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T07:40:00.457 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:00 smithi136 bash[65353]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6 2024-02-10T07:40:00.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:00 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:40:00.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:00 smithi136 ceph-mon[57888]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T07:40:00.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:00 smithi136 ceph-mon[57888]: failed to encode map e136 with expected crc 2024-02-10T07:40:00.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:00 smithi136 ceph-mon[57888]: Health detail: HEALTH_WARN 1 osds down 2024-02-10T07:40:00.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:00 smithi136 ceph-mon[57888]: [WRN] OSD_DOWN: 1 osds down 2024-02-10T07:40:00.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:00 smithi136 ceph-mon[57888]: osd.6 (root=default,host=smithi136) is down 2024-02-10T07:40:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:40:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[64470]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T07:40:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[64470]: failed to encode map e136 with expected crc 2024-02-10T07:40:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[64470]: Health detail: HEALTH_WARN 1 osds down 2024-02-10T07:40:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[64470]: [WRN] OSD_DOWN: 1 osds down 2024-02-10T07:40:00.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[64470]: osd.6 (root=default,host=smithi136) is down 2024-02-10T07:40:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:40:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[67016]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T07:40:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[67016]: failed to encode map e136 with expected crc 2024-02-10T07:40:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[67016]: Health detail: HEALTH_WARN 1 osds down 2024-02-10T07:40:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[67016]: [WRN] OSD_DOWN: 1 osds down 2024-02-10T07:40:00.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:00 smithi064 ceph-mon[67016]: osd.6 (root=default,host=smithi136) is down 2024-02-10T07:40:00.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:00 smithi136 podman[65370]: 2024-02-10 07:40:00.456621622 +0000 UTC m=+0.669847533 container remove 03001fcbe817e9a6fd93aa373ee7b00ee6369ef4f22ce7c49038056a7ff10f7c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True, architecture=x86_64, io.openshift.expose-services=, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git) 2024-02-10T07:40:01.126 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:00 smithi136 podman[65464]: 2024-02-10 07:40:00.687483761 +0000 UTC m=+0.018589420 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:40:01.126 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:00 smithi136 podman[65464]: 2024-02-10 07:40:00.790964635 +0000 UTC m=+0.122070290 container create c09b906e90347a654547723cefe865123df1b3fc7bab7ec610e559d189af2b3e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T07:40:01.420 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:01 smithi136 podman[65464]: 2024-02-10 07:40:01.12705774 +0000 UTC m=+0.458163395 container init c09b906e90347a654547723cefe865123df1b3fc7bab7ec610e559d189af2b3e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:40:01.420 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:01 smithi136 podman[65464]: 2024-02-10 07:40:01.130135803 +0000 UTC m=+0.461241467 container start c09b906e90347a654547723cefe865123df1b3fc7bab7ec610e559d189af2b3e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T07:40:01.420 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:01 smithi136 podman[65464]: 2024-02-10 07:40:01.235080899 +0000 UTC m=+0.566186557 container attach c09b906e90347a654547723cefe865123df1b3fc7bab7ec610e559d189af2b3e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:40:01.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: pgmap v177: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:40:01.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: osdmap e137: 8 total, 7 up, 8 in 2024-02-10T07:40:01.421 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.421 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.421 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:01 smithi136 ceph-mon[57888]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: pgmap v177: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:40:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: osdmap e137: 8 total, 7 up, 8 in 2024-02-10T07:40:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[64470]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: pgmap v177: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: osdmap e137: 8 total, 7 up, 8 in 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:01 smithi064 ceph-mon[67016]: failed to encode map e137 with expected crc 2024-02-10T07:40:01.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:01 smithi136 podman[65464]: 2024-02-10 07:40:01.420251501 +0000 UTC m=+0.751357172 container died c09b906e90347a654547723cefe865123df1b3fc7bab7ec610e559d189af2b3e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T07:40:02.471 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 podman[65489]: 2024-02-10 07:40:02.022661422 +0000 UTC m=+0.597078092 container remove c09b906e90347a654547723cefe865123df1b3fc7bab7ec610e559d189af2b3e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T07:40:02.471 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-10T07:40:02.471 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 systemd[1]: Stopped Ceph osd.6 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:40:02.471 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.6.service: Consumed 8.269s CPU time. 2024-02-10T07:40:02.471 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 systemd[1]: Starting Ceph osd.6 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:40:02.717 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:40:02.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 podman[65595]: 2024-02-10 07:40:02.374109628 +0000 UTC m=+0.018021511 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:40:02.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 podman[65595]: 2024-02-10 07:40:02.490294606 +0000 UTC m=+0.134206472 container create 05e43ff8301db7c90593d9409e4498bdc28b60791de3dd1af4a0c5bd14cc0ffd (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T07:40:02.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 podman[65595]: 2024-02-10 07:40:02.701177448 +0000 UTC m=+0.345089317 container init 05e43ff8301db7c90593d9409e4498bdc28b60791de3dd1af4a0c5bd14cc0ffd (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-10T07:40:02.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 podman[65595]: 2024-02-10 07:40:02.704192374 +0000 UTC m=+0.348104240 container start 05e43ff8301db7c90593d9409e4498bdc28b60791de3dd1af4a0c5bd14cc0ffd (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:40:03.140 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:02 smithi136 podman[65595]: 2024-02-10 07:40:02.809651951 +0000 UTC m=+0.453563831 container attach 05e43ff8301db7c90593d9409e4498bdc28b60791de3dd1af4a0c5bd14cc0ffd (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (6m) 99s ago 11m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (5m) 29s ago 11m 54.4M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (5m) 99s ago 10m 54.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (5m) 29s ago 15m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (5m) 99s ago 17m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (5m) 99s ago 17m 49.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (4m) 29s ago 16m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (4m) 99s ago 16m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (6m) 99s ago 12m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (6m) 29s ago 12m 17.0M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (3m) 99s ago 15m 44.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:40:03.387 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (2m) 99s ago 15m 44.0M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (2m) 99s ago 14m 40.7M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (104s) 99s ago 14m 12.3M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (69s) 29s ago 14m 47.0M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (34s) 29s ago 13m 12.2M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 f694bfa437e7 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (13m) 29s ago 13m 58.7M 2992M 17.2.0 e1d6a67b021e 03001fcbe817 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (12m) 29s ago 12m 64.2M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (5m) 29s ago 12m 68.4M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (10m) 99s ago 10m 74.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (10m) 29s ago 10m 73.3M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (10m) 99s ago 10m 73.1M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:40:03.388 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (10m) 29s ago 10m 73.3M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:40:03.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-mon[57888]: pgmap v179: 161 pgs: 13 stale+active+clean, 9 peering, 139 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:40:03.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T07:40:03.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T07:40:03.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T07:40:03.474 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T07:40:03.474 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T07:40:03.474 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T07:40:03.474 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T07:40:03.474 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T07:40:03.474 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 bash[65595]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T07:40:03.475 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T07:40:03.475 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T07:40:03.475 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T07:40:03.475 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T07:40:03.475 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate[65610]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T07:40:03.475 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 podman[65595]: 2024-02-10 07:40:03.165593668 +0000 UTC m=+0.809505542 container died 05e43ff8301db7c90593d9409e4498bdc28b60791de3dd1af4a0c5bd14cc0ffd (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T07:40:03.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:03 smithi064 ceph-mon[64470]: pgmap v179: 161 pgs: 13 stale+active+clean, 9 peering, 139 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:40:03.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:03 smithi064 ceph-mon[67016]: pgmap v179: 161 pgs: 13 stale+active+clean, 9 peering, 139 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:40:03.794 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 11 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:40:03.795 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:40:03.973 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 podman[65743]: 2024-02-10 07:40:03.780523558 +0000 UTC m=+0.609706311 container remove 05e43ff8301db7c90593d9409e4498bdc28b60791de3dd1af4a0c5bd14cc0ffd (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T07:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "11/23 daemons upgraded", 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:40:04.316 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:03 smithi136 podman[65809]: 2024-02-10 07:40:03.900940206 +0000 UTC m=+0.017943358 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:40:04.316 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 podman[65809]: 2024-02-10 07:40:04.023341542 +0000 UTC m=+0.140344691 container create 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T07:40:04.317 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 podman[65809]: 2024-02-10 07:40:04.225920001 +0000 UTC m=+0.342923151 container init 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T07:40:04.317 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 podman[65809]: 2024-02-10 07:40:04.228828476 +0000 UTC m=+0.345831628 container start 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T07:40:04.317 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 ceph-osd[65828]: -- 172.21.15.136:0/3522661689 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ecd27431e0 con 0x55ecd2715000 2024-02-10T07:40:04.549 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 25/627 objects degraded (3.987%), 8 pgs degraded 2024-02-10T07:40:04.549 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T07:40:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: osd.6 (root=default,host=smithi136) is down 2024-02-10T07:40:04.549 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 25/627 objects degraded (3.987%), 8 pgs degraded 2024-02-10T07:40:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-02-10T07:40:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-02-10T07:40:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-02-10T07:40:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-02-10T07:40:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-02-10T07:40:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-02-10T07:40:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-02-10T07:40:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-02-10T07:40:04.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:04 smithi064 ceph-mon[64470]: from='client.44466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:04.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:04 smithi064 ceph-mon[64470]: from='client.54427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:04.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:04 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2662802688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:04 smithi064 ceph-mon[67016]: from='client.44466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:04 smithi064 ceph-mon[67016]: from='client.54427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:04.552 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:04 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2662802688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:04.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:04 smithi136 ceph-mon[57888]: from='client.44466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:04.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:04 smithi136 ceph-mon[57888]: from='client.54427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:04.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:04 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2662802688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:04.728 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 bash[65809]: 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a 2024-02-10T07:40:04.728 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 systemd[1]: Started Ceph osd.6 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:40:05.147 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:40:04.772+0000 7fc41a9be7c0 -1 Falling back to public interface 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: pgmap v180: 161 pgs: 8 stale+active+clean, 8 active+undersized+degraded, 9 peering, 15 active+undersized, 121 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:05.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:05 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/3955598342' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:40:05.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: pgmap v180: 161 pgs: 8 stale+active+clean, 8 active+undersized+degraded, 9 peering, 15 active+undersized, 121 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3955598342' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:40:05.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:05.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: pgmap v180: 161 pgs: 8 stale+active+clean, 8 active+undersized+degraded, 9 peering, 15 active+undersized, 121 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-02-10T07:40:05.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:05.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:05.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:05.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:05.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:05 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/3955598342' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:40:06.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:06 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:06.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:06 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:06.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:06 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:07.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:07 smithi136 ceph-mon[57888]: pgmap v181: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-10T07:40:07.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:40:07.473+0000 7fc41a9be7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-10T07:40:07.723 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:40:07.500+0000 7fc41a9be7c0 -1 osd.6 135 log_to_monitors true 2024-02-10T07:40:07.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:07 smithi064 ceph-mon[64470]: pgmap v181: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-10T07:40:07.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:07 smithi064 ceph-mon[67016]: pgmap v181: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-10T07:40:07.815 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:40:07] "GET /metrics HTTP/1.1" 200 38527 "" "Prometheus/2.33.4" 2024-02-10T07:40:08.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:08 smithi136 ceph-mon[57888]: from='osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:40:08.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:08 smithi136 ceph-mon[57888]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:40:08.723 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:40:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:40:08.352+0000 7fc40be16700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:40:08.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:08 smithi064 ceph-mon[64470]: from='osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:40:08.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:08 smithi064 ceph-mon[64470]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:40:08.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:08 smithi064 ceph-mon[67016]: from='osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:40:08.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:08 smithi064 ceph-mon[67016]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:40:09.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: pgmap v182: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-10T07:40:09.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:40:09.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: failed to encode map e138 with expected crc 2024-02-10T07:40:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: from='osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: osdmap e138: 8 total, 7 up, 8 in 2024-02-10T07:40:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:09.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:09 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: pgmap v182: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: failed to encode map e138 with expected crc 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: from='osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: osdmap e138: 8 total, 7 up, 8 in 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: pgmap v182: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: failed to encode map e138 with expected crc 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: from='osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: osdmap e138: 8 total, 7 up, 8 in 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:09.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:09 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: failed to encode map e139 with expected crc 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787] boot 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: failed to encode map e139 with expected crc 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:10.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:10 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: failed to encode map e139 with expected crc 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787] boot 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: failed to encode map e139 with expected crc 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:40:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: failed to encode map e139 with expected crc 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: osd.6 [v2:172.21.15.136:6816/454638787,v1:172.21.15.136:6817/454638787] boot 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: failed to encode map e139 with expected crc 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:10.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:10 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:11.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: pgmap v185: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:40:11.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T07:40:11.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:11.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:11 smithi136 ceph-mon[57888]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: pgmap v185: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:40:11.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[64470]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: pgmap v185: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: failed to encode map e140 with expected crc 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:11.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:11 smithi064 ceph-mon[67016]: failed to encode map e140 with expected crc 2024-02-10T07:40:12.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:12 smithi136 ceph-mon[57888]: pgmap v187: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-02-10T07:40:12.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:12 smithi064 ceph-mon[64470]: pgmap v187: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-02-10T07:40:12.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:12 smithi064 ceph-mon[67016]: pgmap v187: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-02-10T07:40:13.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:40:13.410+0000 7f6df9319700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:40:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:15 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:15 smithi064 ceph-mon[64470]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:40:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:15 smithi064 ceph-mon[64470]: pgmap v188: 161 pgs: 9 peering, 4 active+undersized+degraded, 9 active+undersized, 139 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 23/627 objects degraded (3.668%) 2024-02-10T07:40:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:15 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:15 smithi064 ceph-mon[67016]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:40:15.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:15 smithi064 ceph-mon[67016]: pgmap v188: 161 pgs: 9 peering, 4 active+undersized+degraded, 9 active+undersized, 139 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 23/627 objects degraded (3.668%) 2024-02-10T07:40:15.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:15 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:15 smithi136 ceph-mon[57888]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-10T07:40:15.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:15 smithi136 ceph-mon[57888]: pgmap v188: 161 pgs: 9 peering, 4 active+undersized+degraded, 9 active+undersized, 139 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 23/627 objects degraded (3.668%) 2024-02-10T07:40:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:16 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:16.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:16 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:16.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:16 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:17.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:40:17] "GET /metrics HTTP/1.1" 200 38544 "" "Prometheus/2.33.4" 2024-02-10T07:40:17.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-mon[67016]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-02-10T07:40:17.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/627 objects degraded (3.668%), 4 pgs degraded) 2024-02-10T07:40:17.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:40:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-mon[64470]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-02-10T07:40:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/627 objects degraded (3.668%), 4 pgs degraded) 2024-02-10T07:40:17.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:17 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:40:17.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:17 smithi136 ceph-mon[57888]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-02-10T07:40:17.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:17 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/627 objects degraded (3.668%), 4 pgs degraded) 2024-02-10T07:40:17.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:17 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:40:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:19 smithi064 ceph-mon[64470]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:40:19.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:19 smithi064 ceph-mon[67016]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:40:19.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:19 smithi136 ceph-mon[57888]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:40:21.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:21 smithi064 ceph-mon[64470]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:40:21.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:21 smithi064 ceph-mon[67016]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:40:21.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:21 smithi136 ceph-mon[57888]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:40:23.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:23 smithi136 ceph-mon[57888]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 0 op/s 2024-02-10T07:40:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:23 smithi064 ceph-mon[64470]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 0 op/s 2024-02-10T07:40:23.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:23 smithi064 ceph-mon[67016]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 0 op/s 2024-02-10T07:40:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:24 smithi136 ceph-mon[57888]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:24.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:24 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:24 smithi064 ceph-mon[64470]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:25.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:24 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:25.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:24 smithi064 ceph-mon[67016]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:25.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:24 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:25.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:25 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:26.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:25 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:26.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:25 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:26.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:26 smithi136 ceph-mon[57888]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:27.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:26 smithi064 ceph-mon[64470]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:27.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:26 smithi064 ceph-mon[67016]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:27.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:40:27] "GET /metrics HTTP/1.1" 200 38544 "" "Prometheus/2.33.4" 2024-02-10T07:40:29.352 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:29 smithi136 ceph-mon[57888]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:29.352 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:29.352 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:29.352 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:29.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[64470]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:29.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:29.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:29.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:29.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[67016]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:29.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:29.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:29.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:30.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:30 smithi136 ceph-mon[57888]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:30.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:30 smithi136 ceph-mon[57888]: Upgrade: osd.7 is safe to restart 2024-02-10T07:40:30.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:30 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:30.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:30 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:40:30.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:30 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:30.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[64470]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:30.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[64470]: Upgrade: osd.7 is safe to restart 2024-02-10T07:40:30.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:30.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:40:30.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:30.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[67016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T07:40:30.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[67016]: Upgrade: osd.7 is safe to restart 2024-02-10T07:40:30.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:30.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:40:30.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:30 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:31.429 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:31 smithi136 ceph-mon[57888]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:31.430 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:31 smithi136 ceph-mon[57888]: Upgrade: Updating osd.7 2024-02-10T07:40:31.430 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:31 smithi136 ceph-mon[57888]: Deploying daemon osd.7 on smithi136 2024-02-10T07:40:31.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:31 smithi064 ceph-mon[64470]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:31.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:31 smithi064 ceph-mon[64470]: Upgrade: Updating osd.7 2024-02-10T07:40:31.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:31 smithi064 ceph-mon[64470]: Deploying daemon osd.7 on smithi136 2024-02-10T07:40:31.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:31 smithi064 ceph-mon[67016]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:31.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:31 smithi064 ceph-mon[67016]: Upgrade: Updating osd.7 2024-02-10T07:40:31.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:31 smithi064 ceph-mon[67016]: Deploying daemon osd.7 on smithi136 2024-02-10T07:40:33.422 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:33 smithi136 ceph-mon[57888]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:33.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:33 smithi064 ceph-mon[64470]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:33.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:33 smithi064 ceph-mon[67016]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:40:33.722 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:33 smithi136 systemd[1]: Stopping Ceph osd.7 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:40:33.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[37671]: 2024-02-10T07:40:33.666+0000 7f59e93e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:40:33.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[37671]: 2024-02-10T07:40:33.666+0000 7f59e93e2700 -1 osd.7 140 *** Got signal Terminated *** 2024-02-10T07:40:33.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:33 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[37671]: 2024-02-10T07:40:33.666+0000 7f59e93e2700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:40:34.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:34 smithi064 ceph-mon[64470]: osd.7 marked itself down and dead 2024-02-10T07:40:34.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:34 smithi064 ceph-mon[67016]: osd.7 marked itself down and dead 2024-02-10T07:40:34.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:34 smithi136 ceph-mon[57888]: osd.7 marked itself down and dead 2024-02-10T07:40:34.911 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:40:35.034 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:34 smithi136 podman[67689]: 2024-02-10 07:40:34.761320668 +0000 UTC m=+1.198498537 container died 199d74b3948c7b8c84ac4de6195dbef8ceb9df06321df814094dc182375ec875 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, release=754, GIT_BRANCH=HEAD, architecture=x86_64, name=centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , distribution-scope=public) 2024-02-10T07:40:35.306 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 podman[67689]: 2024-02-10 07:40:35.091524968 +0000 UTC m=+1.528702828 container cleanup 199d74b3948c7b8c84ac4de6195dbef8ceb9df06321df814094dc182375ec875 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T07:40:35.306 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 bash[67689]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7 2024-02-10T07:40:35.306 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 podman[67704]: 2024-02-10 07:40:35.276193855 +0000 UTC m=+0.512311013 container remove 199d74b3948c7b8c84ac4de6195dbef8ceb9df06321df814094dc182375ec875 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_CLEAN=True, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-10T07:40:35.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[67016]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[67016]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[67016]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[67016]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[64470]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[64470]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[64470]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T07:40:35.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:35 smithi064 ceph-mon[64470]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T07:40:35.589 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:40:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (7m) 2m ago 11m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:40:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (6m) 26s ago 11m 54.7M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:40:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (5m) 2m ago 10m 54.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:40:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (5m) 26s ago 16m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:40:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (6m) 2m ago 18m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:40:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (5m) 2m ago 18m 49.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (4m) 26s ago 16m 41.3M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (5m) 2m ago 16m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (7m) 2m ago 12m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (6m) 26s ago 12m 17.0M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (4m) 2m ago 16m 44.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (3m) 2m ago 15m 44.0M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (2m) 2m ago 15m 40.7M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (2m) 2m ago 14m 12.3M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:40:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (101s) 26s ago 14m 50.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (66s) 26s ago 14m 53.2M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 f694bfa437e7 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (31s) 26s ago 13m 12.7M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 59163ab2c153 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (13m) 26s ago 13m 66.8M 2992M 17.2.0 e1d6a67b021e 199d74b3948c 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (5m) 26s ago 12m 68.5M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (11m) 2m ago 11m 74.4M - 17.2.0 e1d6a67b021e a5a3ae19d92e 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (11m) 26s ago 11m 73.5M - 17.2.0 e1d6a67b021e 51b5a68fefad 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (11m) 2m ago 11m 73.1M - 17.2.0 e1d6a67b021e 9e11c8d46b85 2024-02-10T07:40:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (10m) 26s ago 10m 73.4M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:40:35.707 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:35 smithi136 ceph-mon[57888]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:35.707 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:35 smithi136 ceph-mon[57888]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:40:35.707 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:35 smithi136 ceph-mon[57888]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T07:40:35.707 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:35 smithi136 ceph-mon[57888]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T07:40:35.966 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 podman[67800]: 2024-02-10 07:40:35.609293425 +0000 UTC m=+0.017968234 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:40:35.967 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 podman[67800]: 2024-02-10 07:40:35.73555241 +0000 UTC m=+0.144227249 container create 10cb471122ef24f76a89cc9beb29c6e1ecce8b4751d7036f3dbc6b9d17f4ddd2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 12 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:40:35.994 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:40:36.224 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 podman[67800]: 2024-02-10 07:40:35.962825594 +0000 UTC m=+0.371500387 container init 10cb471122ef24f76a89cc9beb29c6e1ecce8b4751d7036f3dbc6b9d17f4ddd2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T07:40:36.224 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:35 smithi136 podman[67800]: 2024-02-10 07:40:35.965886166 +0000 UTC m=+0.374560964 container start 10cb471122ef24f76a89cc9beb29c6e1ecce8b4751d7036f3dbc6b9d17f4ddd2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T07:40:36.224 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:36 smithi136 podman[67800]: 2024-02-10 07:40:36.046243689 +0000 UTC m=+0.454918499 container attach 10cb471122ef24f76a89cc9beb29c6e1ecce8b4751d7036f3dbc6b9d17f4ddd2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:40:36.334 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:40:36.334 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "12/23 daemons upgraded", 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:40:36.335 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:40:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[67016]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[67016]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[67016]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T07:40:36.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2538153890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:36.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[64470]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[64470]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[64470]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T07:40:36.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:36 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2538153890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:36.575 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:36 smithi136 podman[67800]: 2024-02-10 07:40:36.256920973 +0000 UTC m=+0.665595767 container died 10cb471122ef24f76a89cc9beb29c6e1ecce8b4751d7036f3dbc6b9d17f4ddd2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T07:40:36.575 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:36 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:36.575 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:36 smithi136 ceph-mon[57888]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:36.575 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:36 smithi136 ceph-mon[57888]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:36.575 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:36 smithi136 ceph-mon[57888]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T07:40:36.575 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:36 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2538153890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:36.747 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid 2024-02-10T07:40:36.747 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T07:40:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: osd.7 (root=default,host=smithi136) is down 2024-02-10T07:40:36.747 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-10T07:40:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-10T07:40:37.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:36 smithi136 podman[67826]: 2024-02-10 07:40:36.900490429 +0000 UTC m=+0.638620911 container remove 10cb471122ef24f76a89cc9beb29c6e1ecce8b4751d7036f3dbc6b9d17f4ddd2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:40:37.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:36 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-10T07:40:37.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:36 smithi136 systemd[1]: Stopped Ceph osd.7 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:40:37.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:36 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.7.service: Consumed 11.063s CPU time. 2024-02-10T07:40:37.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:37 smithi136 systemd[1]: Starting Ceph osd.7 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:40:37.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:40:37] "GET /metrics HTTP/1.1" 200 38542 "" "Prometheus/2.33.4" 2024-02-10T07:40:37.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[64470]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:40:37.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[64470]: from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:37.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[64470]: from='client.54487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:37.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/4063048270' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:40:37.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[67016]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:40:37.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[67016]: from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:37.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[67016]: from='client.54487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:37.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:37 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/4063048270' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:40:37.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:37 smithi136 ceph-mon[57888]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:40:37.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:37 smithi136 ceph-mon[57888]: from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:37.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:37 smithi136 ceph-mon[57888]: from='client.54487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:40:37.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:37 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/4063048270' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:40:37.596 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:37 smithi136 podman[67929]: 2024-02-10 07:40:37.239283944 +0000 UTC m=+0.018788975 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:40:37.596 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:37 smithi136 podman[67929]: 2024-02-10 07:40:37.359866478 +0000 UTC m=+0.139371538 container create 82e650ee9897e6ce701a16c831f33fbbb788af738dfc61ff0a1bf186cfc33b41 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:40:37.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:37 smithi136 podman[67929]: 2024-02-10 07:40:37.595483298 +0000 UTC m=+0.374988330 container init 82e650ee9897e6ce701a16c831f33fbbb788af738dfc61ff0a1bf186cfc33b41 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T07:40:37.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:37 smithi136 podman[67929]: 2024-02-10 07:40:37.598744987 +0000 UTC m=+0.378250006 container start 82e650ee9897e6ce701a16c831f33fbbb788af738dfc61ff0a1bf186cfc33b41 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T07:40:37.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:37 smithi136 podman[67929]: 2024-02-10 07:40:37.678893443 +0000 UTC m=+0.458398469 container attach 82e650ee9897e6ce701a16c831f33fbbb788af738dfc61ff0a1bf186cfc33b41 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a) 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 bash[67929]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T07:40:38.473 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T07:40:38.474 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T07:40:38.474 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T07:40:38.474 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate[67944]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T07:40:38.474 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 podman[67929]: 2024-02-10 07:40:38.070998489 +0000 UTC m=+0.850503516 container died 82e650ee9897e6ce701a16c831f33fbbb788af738dfc61ff0a1bf186cfc33b41 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T07:40:39.127 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:38 smithi136 podman[68077]: 2024-02-10 07:40:38.799846325 +0000 UTC m=+0.723207028 container remove 82e650ee9897e6ce701a16c831f33fbbb788af738dfc61ff0a1bf186cfc33b41 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T07:40:39.127 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 podman[68142]: 2024-02-10 07:40:38.918069192 +0000 UTC m=+0.018479406 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:40:39.127 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 podman[68142]: 2024-02-10 07:40:39.025735918 +0000 UTC m=+0.126146117 container create a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T07:40:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:39 smithi064 ceph-mon[64470]: pgmap v202: 161 pgs: 11 active+undersized+degraded, 11 active+undersized, 139 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-02-10T07:40:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:39 smithi064 ceph-mon[64470]: Health check failed: Degraded data redundancy: 34/627 objects degraded (5.423%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:39 smithi064 ceph-mon[67016]: pgmap v202: 161 pgs: 11 active+undersized+degraded, 11 active+undersized, 139 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-02-10T07:40:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:39 smithi064 ceph-mon[67016]: Health check failed: Degraded data redundancy: 34/627 objects degraded (5.423%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:39.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:39 smithi136 ceph-mon[57888]: pgmap v202: 161 pgs: 11 active+undersized+degraded, 11 active+undersized, 139 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-02-10T07:40:39.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:39 smithi136 ceph-mon[57888]: Health check failed: Degraded data redundancy: 34/627 objects degraded (5.423%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:39.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 podman[68142]: 2024-02-10 07:40:39.236625522 +0000 UTC m=+0.337035726 container init a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T07:40:39.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 podman[68142]: 2024-02-10 07:40:39.239505864 +0000 UTC m=+0.339916068 container start a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T07:40:39.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 ceph-osd[68161]: -- 172.21.15.136:0/881691266 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6408e91e0 con 0x55e6408bb400 2024-02-10T07:40:39.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 bash[68142]: a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 2024-02-10T07:40:39.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 systemd[1]: Started Ceph osd.7 for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:40:40.157 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:39 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:40:39.777+0000 7f9f8b0557c0 -1 Falling back to public interface 2024-02-10T07:40:40.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:40.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:40 smithi136 ceph-mon[57888]: pgmap v203: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:40.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:40.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:40.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:40 smithi064 ceph-mon[67016]: pgmap v203: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:40.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:40.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:40.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:40 smithi064 ceph-mon[64470]: pgmap v203: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:42.625 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:42 smithi136 ceph-mon[57888]: pgmap v204: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:42.625 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:40:42.507+0000 7f9f8b0557c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-10T07:40:42.625 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:42 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:40:42.539+0000 7f9f8b0557c0 -1 osd.7 140 log_to_monitors true 2024-02-10T07:40:42.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:42 smithi064 ceph-mon[64470]: pgmap v204: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:42.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:42 smithi064 ceph-mon[67016]: pgmap v204: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:43.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[67016]: from='osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:40:43.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[67016]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:40:43.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:43.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:43.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[64470]: from='osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:40:43.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[64470]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:40:43.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:43.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:43 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:43.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:43 smithi136 ceph-mon[57888]: from='osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:40:43.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:43 smithi136 ceph-mon[57888]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:40:43.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:43.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:43 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:44.222 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:40:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:40:43.821+0000 7f9f784a5700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:40:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: pgmap v205: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:40:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: from='osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T07:40:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[64470]: from='osd.7 ' entity='osd.7' 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: pgmap v205: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: from='osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:44.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:44 smithi064 ceph-mon[67016]: from='osd.7 ' entity='osd.7' 2024-02-10T07:40:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: pgmap v205: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:40:44.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: from='osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T07:40:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-02-10T07:40:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:44.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:44 smithi136 ceph-mon[57888]: from='osd.7 ' entity='osd.7' 2024-02-10T07:40:45.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:45.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:45.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:45.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:40:45.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636] boot 2024-02-10T07:40:45.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T07:40:45.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:45 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636] boot 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T07:40:46.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: osd.7 [v2:172.21.15.136:6824/1523823636,v1:172.21.15.136:6825/1523823636] boot 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T07:40:46.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:45 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:40:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:46 smithi136 ceph-mon[57888]: pgmap v208: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:46 smithi136 ceph-mon[57888]: osdmap e145: 8 total, 8 up, 8 in 2024-02-10T07:40:47.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:46 smithi064 ceph-mon[64470]: pgmap v208: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:47.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:46 smithi064 ceph-mon[64470]: osdmap e145: 8 total, 8 up, 8 in 2024-02-10T07:40:47.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:46 smithi064 ceph-mon[67016]: pgmap v208: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-10T07:40:47.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:46 smithi064 ceph-mon[67016]: osdmap e145: 8 total, 8 up, 8 in 2024-02-10T07:40:47.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:40:47] "GET /metrics HTTP/1.1" 200 38625 "" "Prometheus/2.33.4" 2024-02-10T07:40:47.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:47 smithi136 ceph-mon[57888]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:48.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:47 smithi064 ceph-mon[64470]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:48.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:47 smithi064 ceph-mon[67016]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: pgmap v210: 161 pgs: 18 active+undersized+degraded, 32 active+undersized, 111 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 60/627 objects degraded (9.569%) 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:49.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all osd 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T07:40:49.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T07:40:49.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:49 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T07:40:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: pgmap v210: 161 pgs: 18 active+undersized+degraded, 32 active+undersized, 111 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 60/627 objects degraded (9.569%) 2024-02-10T07:40:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all osd 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T07:40:49.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T07:40:49.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: pgmap v210: 161 pgs: 18 active+undersized+degraded, 32 active+undersized, 111 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 60/627 objects degraded (9.569%) 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all osd 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T07:40:49.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T07:40:49.569 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:49 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T07:40:50.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[64470]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T07:40:50.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T07:40:50.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[64470]: osdmap e146: 8 total, 8 up, 8 in 2024-02-10T07:40:50.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all mds 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[67016]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[67016]: osdmap e146: 8 total, 8 up, 8 in 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all mds 2024-02-10T07:40:50.257 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:50 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:50.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:50 smithi136 ceph-mon[57888]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T07:40:50.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:50 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T07:40:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:50 smithi136 ceph-mon[57888]: osdmap e146: 8 total, 8 up, 8 in 2024-02-10T07:40:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:50 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:40:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:50 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all mds 2024-02-10T07:40:50.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:50 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[67016]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-02-10T07:40:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[67016]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded) 2024-02-10T07:40:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[67016]: Cluster is now healthy 2024-02-10T07:40:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:40:51.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:51.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[64470]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-02-10T07:40:51.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[64470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded) 2024-02-10T07:40:51.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[64470]: Cluster is now healthy 2024-02-10T07:40:51.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:51.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:40:51.315 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:51 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:51.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:51 smithi136 ceph-mon[57888]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-02-10T07:40:51.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:51 smithi136 ceph-mon[57888]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded) 2024-02-10T07:40:51.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:51 smithi136 ceph-mon[57888]: Cluster is now healthy 2024-02-10T07:40:51.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:51 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:51.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:51 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi064.elxhce", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:40:51.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:51 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:52.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:52 smithi064 ceph-mon[64470]: Upgrade: Updating rgw.foo.smithi064.elxhce (1/4) 2024-02-10T07:40:52.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:52 smithi064 ceph-mon[64470]: Deploying daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:40:52.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:52 smithi064 ceph-mon[67016]: Upgrade: Updating rgw.foo.smithi064.elxhce (1/4) 2024-02-10T07:40:52.315 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:52 smithi064 ceph-mon[67016]: Deploying daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:40:52.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:52 smithi136 ceph-mon[57888]: Upgrade: Updating rgw.foo.smithi064.elxhce (1/4) 2024-02-10T07:40:52.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:52 smithi136 ceph-mon[57888]: Deploying daemon rgw.foo.smithi064.elxhce on smithi064 2024-02-10T07:40:53.213 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:53 smithi064 ceph-mon[67016]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-02-10T07:40:53.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:53 smithi064 ceph-mon[64470]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-02-10T07:40:53.392 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:53 smithi136 ceph-mon[57888]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-02-10T07:40:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:55 smithi064 ceph-mon[67016]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:55.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:55 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:55 smithi064 ceph-mon[64470]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:55.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:55 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:55.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:55 smithi136 ceph-mon[57888]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:40:55.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:55.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:55 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:56.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:56 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:56.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:56 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:56 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:57.534 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:40:57] "GET /metrics HTTP/1.1" 200 38625 "" "Prometheus/2.33.4" 2024-02-10T07:40:57.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[64470]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:40:57.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:57.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:40:57.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:57.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[67016]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:40:57.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:57.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:40:57.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:57 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:57.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:57 smithi136 ceph-mon[57888]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:40:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi064.dqyqnr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:40:57.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:57 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:58.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:58 smithi064 ceph-mon[64470]: Upgrade: Updating rgw.smpl.smithi064.dqyqnr (2/4) 2024-02-10T07:40:58.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:58 smithi064 ceph-mon[64470]: Deploying daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:40:58.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:58 smithi064 ceph-mon[67016]: Upgrade: Updating rgw.smpl.smithi064.dqyqnr (2/4) 2024-02-10T07:40:58.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:58 smithi064 ceph-mon[67016]: Deploying daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:40:58.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:58 smithi136 ceph-mon[57888]: Upgrade: Updating rgw.smpl.smithi064.dqyqnr (2/4) 2024-02-10T07:40:58.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:58 smithi136 ceph-mon[57888]: Deploying daemon rgw.smpl.smithi064.dqyqnr on smithi064 2024-02-10T07:40:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[64470]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-02-10T07:40:59.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:59.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:59.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[67016]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-02-10T07:40:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:59.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:40:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:59 smithi136 ceph-mon[57888]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-02-10T07:40:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:40:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:59.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:40:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:01 smithi064 ceph-mon[64470]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 101 B/s wr, 100 op/s 2024-02-10T07:41:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:01 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:01.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:01 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:01 smithi064 ceph-mon[67016]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 101 B/s wr, 100 op/s 2024-02-10T07:41:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:01 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:01.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:01 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:01.568 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:01 smithi136 ceph-mon[57888]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 101 B/s wr, 100 op/s 2024-02-10T07:41:01.569 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:01 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:01.569 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:01 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:03.428 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:03 smithi136 ceph-mon[57888]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-10T07:41:03.428 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:03 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:03.428 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:03 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:41:03.429 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:03 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:03.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[64470]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-10T07:41:03.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:03.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:41:03.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:03.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[67016]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-10T07:41:03.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:03.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi136.xkhtnu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:41:03.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:03 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:04.596 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:04 smithi136 ceph-mon[57888]: Upgrade: Updating rgw.foo.smithi136.xkhtnu (3/4) 2024-02-10T07:41:04.596 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:04 smithi136 ceph-mon[57888]: Deploying daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:41:04.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:04 smithi064 ceph-mon[64470]: Upgrade: Updating rgw.foo.smithi136.xkhtnu (3/4) 2024-02-10T07:41:04.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:04 smithi064 ceph-mon[64470]: Deploying daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:41:04.637 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:04 smithi064 ceph-mon[67016]: Upgrade: Updating rgw.foo.smithi136.xkhtnu (3/4) 2024-02-10T07:41:04.638 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:04 smithi064 ceph-mon[67016]: Deploying daemon rgw.foo.smithi136.xkhtnu on smithi136 2024-02-10T07:41:05.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:05 smithi136 ceph-mon[57888]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 129 op/s 2024-02-10T07:41:05.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:05 smithi064 ceph-mon[64470]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 129 op/s 2024-02-10T07:41:05.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:05 smithi064 ceph-mon[67016]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 129 op/s 2024-02-10T07:41:06.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:06 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:06.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:06 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:06.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:06 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:06 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:06 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:06 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:06.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:06 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:06.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:06 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:06.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:06 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:07.103 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:41:07.418 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:07 smithi064 ceph-mon[64470]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:07.418 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:07 smithi064 ceph-mon[67016]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:07.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:07 smithi136 ceph-mon[57888]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:07.771 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:41:07] "GET /metrics HTTP/1.1" 200 38631 "" "Prometheus/2.33.4" 2024-02-10T07:41:07.771 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:41:07.771 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (7m) 2m ago 12m 24.7M - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:41:07.771 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (7m) 24s ago 12m 55.2M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:41:07.771 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 running (6m) 2m ago 11m 54.5M - 3.5 e1d6a67b021e 41a9cb2c8a65 2024-02-10T07:41:07.771 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (6m) 24s ago 16m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (6m) 2m ago 18m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (6m) 2m ago 18m 49.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (5m) 24s ago 17m 41.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (5m) 2m ago 17m 40.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (7m) 2m ago 13m 16.9M - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (7m) 24s ago 13m 16.8M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (4m) 2m ago 16m 44.2M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (3m) 2m ago 16m 44.0M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (3m) 2m ago 15m 40.7M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (2m) 2m ago 15m 12.3M 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (2m) 24s ago 15m 52.5M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (99s) 24s ago 14m 54.8M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 f694bfa437e7 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (63s) 24s ago 14m 43.3M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 59163ab2c153 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (28s) 24s ago 13m 13.6M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 a997551fd755 2024-02-10T07:41:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (6m) 24s ago 13m 68.5M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:41:07.773 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 starting - - - - 2024-02-10T07:41:07.773 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 starting - - - - 2024-02-10T07:41:07.773 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 starting - - - - 2024-02-10T07:41:07.773 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (11m) 24s ago 11m 73.6M - 17.2.0 e1d6a67b021e 2f0084beefed 2024-02-10T07:41:08.180 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:41:08.180 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:41:08.180 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:41:08.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:08.180 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:41:08.180 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T07:41:08.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 15 2024-02-10T07:41:08.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:41:08.182 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:41:08.530 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:08 smithi136 ceph-mon[57888]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:08.530 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:08 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:08.530 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:08 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:41:08.530 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:08 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:08.530 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:08 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/435587283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:08.533 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:41:08.533 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:41:08.533 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:41:08.533 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:41:08.533 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: "osd", 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "13/23 daemons upgraded", 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Currently upgrading rgw daemons", 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": false 2024-02-10T07:41:08.534 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:41:08.534 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[67016]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:08.534 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:08.534 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/435587283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[64470]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi136.fharky", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:08.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:08 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/435587283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:08.939 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_OK 2024-02-10T07:41:09.596 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:09 smithi136 ceph-mon[57888]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:09.596 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:09 smithi136 ceph-mon[57888]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-02-10T07:41:09.596 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:09 smithi136 ceph-mon[57888]: from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:09.597 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:09 smithi136 ceph-mon[57888]: Upgrade: Updating rgw.smpl.smithi136.fharky (4/4) 2024-02-10T07:41:09.597 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:09 smithi136 ceph-mon[57888]: Deploying daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:41:09.597 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:09 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/800959792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:41:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[67016]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[67016]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-02-10T07:41:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[67016]: from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[67016]: Upgrade: Updating rgw.smpl.smithi136.fharky (4/4) 2024-02-10T07:41:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[67016]: Deploying daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:41:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/800959792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:41:09.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[64470]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:09.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[64470]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-02-10T07:41:09.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[64470]: from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:09.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[64470]: Upgrade: Updating rgw.smpl.smithi136.fharky (4/4) 2024-02-10T07:41:09.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[64470]: Deploying daemon rgw.smpl.smithi136.fharky on smithi136 2024-02-10T07:41:09.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:09 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/800959792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:41:10.644 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:10 smithi136 ceph-mon[57888]: from='client.54583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:10 smithi064 ceph-mon[64470]: from='client.54583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:10 smithi064 ceph-mon[67016]: from='client.54583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:11.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:11 smithi136 ceph-mon[57888]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 255 B/s wr, 225 op/s 2024-02-10T07:41:11.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:11 smithi064 ceph-mon[64470]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 255 B/s wr, 225 op/s 2024-02-10T07:41:11.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:11 smithi064 ceph-mon[67016]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 255 B/s wr, 225 op/s 2024-02-10T07:41:13.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:13 smithi136 ceph-mon[57888]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:13.222 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:13 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:13.223 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:13 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:13.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:13 smithi064 ceph-mon[67016]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:13.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:13 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:13.314 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:13 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:13.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:13 smithi064 ceph-mon[64470]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:13.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:13 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:13.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:13 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:14.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:14.631 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:15.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:15 smithi064 ceph-mon[64470]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 213 op/s 2024-02-10T07:41:15.404 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:15 smithi064 ceph-mon[67016]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 213 op/s 2024-02-10T07:41:15.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:15 smithi136 ceph-mon[57888]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 213 op/s 2024-02-10T07:41:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:16 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:16 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:16 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:16 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:16 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:16.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:16 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[64470]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 255 B/s wr, 207 op/s 2024-02-10T07:41:17.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[67016]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 255 B/s wr, 207 op/s 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.420 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:17 smithi136 ceph-mon[57888]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 255 B/s wr, 207 op/s 2024-02-10T07:41:17.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:17 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:17.693 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:41:17] "GET /metrics HTTP/1.1" 200 38619 "" "Prometheus/2.33.4" 2024-02-10T07:41:19.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:19 smithi064 ceph-mon[64470]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:19.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:19 smithi064 ceph-mon[67016]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:19.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:19 smithi136 ceph-mon[57888]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-10T07:41:21.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-02-10T07:41:21.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: Detected new or changed devices on smithi136 2024-02-10T07:41:21.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.566 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi064.elxhce"}]: dispatch 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi064.elxhce"}]': finished 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi136.xkhtnu"}]: dispatch 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi136.xkhtnu"}]': finished 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi064.dqyqnr"}]: dispatch 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi064.dqyqnr"}]': finished 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi136.fharky"}]: dispatch 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi136.fharky"}]': finished 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.567 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: Detected new or changed devices on smithi136 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.568 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi064.elxhce"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi064.elxhce"}]': finished 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi136.xkhtnu"}]: dispatch 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi136.xkhtnu"}]': finished 2024-02-10T07:41:21.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi064.dqyqnr"}]: dispatch 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi064.dqyqnr"}]': finished 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi136.fharky"}]: dispatch 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi136.fharky"}]': finished 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:21 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: Detected new or changed devices on smithi136 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:21.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi064.elxhce"}]: dispatch 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi064.elxhce"}]': finished 2024-02-10T07:41:21.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi136.xkhtnu"}]: dispatch 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi136.xkhtnu"}]': finished 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi064.dqyqnr"}]: dispatch 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi064.dqyqnr"}]': finished 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi136.fharky"}]: dispatch 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi136.fharky"}]': finished 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:21.725 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:21 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:22.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: Detected new or changed devices on smithi064 2024-02-10T07:41:22.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all rgw 2024-02-10T07:41:22.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T07:41:22.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T07:41:22.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: Detected new or changed devices on smithi064 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all rgw 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:41:22.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:22 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:22.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: Detected new or changed devices on smithi064 2024-02-10T07:41:22.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all rgw 2024-02-10T07:41:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T07:41:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T07:41:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi064.yotmxt", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:41:22.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:22 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:23 smithi064 ceph-mon[64470]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-10T07:41:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:23 smithi064 ceph-mon[64470]: Upgrade: Updating iscsi.foo.smithi064.yotmxt 2024-02-10T07:41:23.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:23 smithi064 ceph-mon[64470]: Deploying daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:41:23.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:23 smithi064 ceph-mon[67016]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-10T07:41:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:23 smithi064 ceph-mon[67016]: Upgrade: Updating iscsi.foo.smithi064.yotmxt 2024-02-10T07:41:23.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:23 smithi064 ceph-mon[67016]: Deploying daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:41:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:23 smithi136 ceph-mon[57888]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-10T07:41:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:23 smithi136 ceph-mon[57888]: Upgrade: Updating iscsi.foo.smithi064.yotmxt 2024-02-10T07:41:23.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:23 smithi136 ceph-mon[57888]: Deploying daemon iscsi.foo.smithi064.yotmxt on smithi064 2024-02-10T07:41:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:25 smithi064 ceph-mon[64470]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-10T07:41:25.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:25 smithi064 ceph-mon[67016]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-10T07:41:25.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:25 smithi136 ceph-mon[57888]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-10T07:41:26.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:26 smithi064 ceph-mon[64470]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:26.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:26 smithi064 ceph-mon[67016]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:26.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:26 smithi136 ceph-mon[57888]: from='client.15369 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:27.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:27 smithi064 ceph-mon[67016]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-02-10T07:41:27.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:27 smithi064 ceph-mon[64470]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-02-10T07:41:27.565 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:41:27] "GET /metrics HTTP/1.1" 200 38619 "" "Prometheus/2.33.4" 2024-02-10T07:41:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:27 smithi136 ceph-mon[57888]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-02-10T07:41:29.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:29 smithi136 ceph-mon[57888]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:29.724 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:29.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:29 smithi064 ceph-mon[67016]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:29.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:29.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:29 smithi064 ceph-mon[64470]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:29.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:31 smithi136 ceph-mon[57888]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:31.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:31 smithi064 ceph-mon[64470]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:31 smithi064 ceph-mon[67016]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:33.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:33 smithi136 ceph-mon[57888]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:33.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:33 smithi064 ceph-mon[67016]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:33.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:33 smithi064 ceph-mon[64470]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:35.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:35 smithi064 ceph-mon[64470]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:35.586 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:35 smithi064 ceph-mon[67016]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:35 smithi136 ceph-mon[57888]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:36.034 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:35 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:41:35.670+0000 7f6dc9ab1700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. 2024-02-10T07:41:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:36 smithi136 ceph-mon[57888]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:36 smithi136 ceph-mon[57888]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. 2024-02-10T07:41:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:36 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:36 smithi136 ceph-mon[57888]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-02-10T07:41:36.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:36 smithi136 ceph-mon[57888]: Health check failed: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-10T07:41:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[64470]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[64470]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. 2024-02-10T07:41:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[64470]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-02-10T07:41:37.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[64470]: Health check failed: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-10T07:41:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[67016]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:41:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[67016]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. 2024-02-10T07:41:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[67016]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-02-10T07:41:37.065 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:36 smithi064 ceph-mon[67016]: Health check failed: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-10T07:41:37.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:41:37] "GET /metrics HTTP/1.1" 200 38694 "" "Prometheus/2.33.4" 2024-02-10T07:41:39.301 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:41:39.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:39 smithi064 ceph-mon[64470]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-02-10T07:41:39.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:39 smithi064 ceph-mon[67016]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-02-10T07:41:39.671 INFO:teuthology.orchestra.run.smithi064.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed." 2024-02-10T07:41:39.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:39 smithi136 ceph-mon[57888]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-02-10T07:41:40.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:40.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.462 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:40.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:40 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T07:41:40.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:40.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:40.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:40 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:41.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: from='client.54613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: Checking dashboard <-> RGW credentials 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 447 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[67016]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: from='client.54613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: Checking dashboard <-> RGW credentials 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-02-10T07:41:41.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 447 B/s rd, 0 op/s 2024-02-10T07:41:41.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:41 smithi064 ceph-mon[64470]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: from='client.54613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 701 B/s rd, 0 op/s 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: Checking dashboard <-> RGW credentials 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 447 B/s rd, 0 op/s 2024-02-10T07:41:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:41 smithi136 ceph-mon[57888]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T07:41:42.448 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:41:42.448 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (8m) 2s ago 12m - - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:41:42.448 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (7m) 3s ago 12m 54.8M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:41:42.448 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 unknown 2s ago 11m - - 2024-02-10T07:41:42.448 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (6m) 3s ago 17m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (7m) 2s ago 19m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (6m) 2s ago 19m - 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (5m) 3s ago 17m 44.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (6m) 2s ago 17m - 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (8m) 2s ago 13m - - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (7m) 3s ago 13m 17.1M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (5m) 2s ago 17m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (4m) 2s ago 16m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (3m) 2s ago 16m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (3m) 2s ago 16m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (2m) 3s ago 15m 58.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (2m) 3s ago 15m 58.7M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 f694bfa437e7 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (98s) 3s ago 14m 47.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 59163ab2c153 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (63s) 3s ago 14m 53.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 a997551fd755 2024-02-10T07:41:42.449 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (7m) 3s ago 13m 70.3M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:41:42.450 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (47s) 2s ago 12m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 78bc43a25dc5 2024-02-10T07:41:42.450 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (36s) 3s ago 12m 77.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 d0ccfe9ecfbe 2024-02-10T07:41:42.450 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (41s) 2s ago 12m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 dbbaac6e1a25 2024-02-10T07:41:42.450 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (30s) 3s ago 12m 74.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 cd5364e8847f 2024-02-10T07:41:43.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T07:41:43.500 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:43 smithi064 ceph-mon[64470]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 321 B/s rd, 0 op/s 2024-02-10T07:41:43.500 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:43 smithi064 ceph-mon[67016]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 321 B/s rd, 0 op/s 2024-02-10T07:41:43.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:43 smithi136 ceph-mon[57888]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 321 B/s rd, 0 op/s 2024-02-10T07:41:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:44 smithi064 ceph-mon[64470]: from='client.54655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:44.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:44 smithi064 ceph-mon[67016]: from='client.54655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:44.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:44 smithi136 ceph-mon[57888]: from='client.54655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:41:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:41:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:41:45.282 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T07:41:45.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:41:45.282 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:41:45.282 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 17 2024-02-10T07:41:45.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:41:45.282 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:41:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:45 smithi064 ceph-mon[64470]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-02-10T07:41:45.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:45 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1455310172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:45.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:45 smithi064 ceph-mon[67016]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-02-10T07:41:45.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:45 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1455310172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:45.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:45 smithi136 ceph-mon[57888]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-02-10T07:41:45.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:45 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1455310172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:41:46.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-10T07:41:47.606 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:47 smithi064 ceph-mon[64470]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-02-10T07:41:47.606 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:47 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3760602231' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:41:47.606 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:41:47] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-02-10T07:41:47.606 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:47 smithi064 ceph-mon[67016]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-02-10T07:41:47.606 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:47 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/3760602231' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:41:47.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:47 smithi136 ceph-mon[57888]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-02-10T07:41:47.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:47 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/3760602231' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:41:48.092 INFO:teuthology.orchestra.run.smithi064.stdout:wait for servicemap items w/ changing names to refresh 2024-02-10T07:41:48.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:48 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/2340702838' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]: dispatch 2024-02-10T07:41:48.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:48 smithi064 ceph-mon[64470]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]: dispatch 2024-02-10T07:41:48.651 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:48 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/2340702838' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]: dispatch 2024-02-10T07:41:48.651 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:48 smithi064 ceph-mon[67016]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]: dispatch 2024-02-10T07:41:48.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:48 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/2340702838' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]: dispatch 2024-02-10T07:41:48.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:48 smithi136 ceph-mon[57888]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]: dispatch 2024-02-10T07:41:48.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 60' 2024-02-10T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[67016]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[67016]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]': finished 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[67016]: osdmap e147: 8 total, 8 up, 8 in 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1438377296' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]: dispatch 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[67016]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]: dispatch 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[64470]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[64470]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]': finished 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[64470]: osdmap e147: 8 total, 8 up, 8 in 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1438377296' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]: dispatch 2024-02-10T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:49 smithi064 ceph-mon[64470]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]: dispatch 2024-02-10T07:41:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:49 smithi136 ceph-mon[57888]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-10T07:41:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:49 smithi136 ceph-mon[57888]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6801/2991743476"}]': finished 2024-02-10T07:41:49.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:49 smithi136 ceph-mon[57888]: osdmap e147: 8 total, 8 up, 8 in 2024-02-10T07:41:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:49 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1438377296' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]: dispatch 2024-02-10T07:41:49.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:49 smithi136 ceph-mon[57888]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]: dispatch 2024-02-10T07:41:50.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:50 smithi064 ceph-mon[64470]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]': finished 2024-02-10T07:41:50.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:50 smithi064 ceph-mon[64470]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T07:41:50.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:50 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/228788979' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1773772302"}]: dispatch 2024-02-10T07:41:50.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:50 smithi064 ceph-mon[67016]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]': finished 2024-02-10T07:41:50.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:50 smithi064 ceph-mon[67016]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T07:41:50.682 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:50 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/228788979' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1773772302"}]: dispatch 2024-02-10T07:41:50.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:50 smithi136 ceph-mon[57888]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1765968155"}]': finished 2024-02-10T07:41:50.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:50 smithi136 ceph-mon[57888]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T07:41:50.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:50 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/228788979' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1773772302"}]: dispatch 2024-02-10T07:41:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:51 smithi136 ceph-mon[57888]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-02-10T07:41:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:51 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/228788979' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1773772302"}]': finished 2024-02-10T07:41:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:51 smithi136 ceph-mon[57888]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T07:41:51.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:51 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/342656354' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]: dispatch 2024-02-10T07:41:51.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:51 smithi136 ceph-mon[57888]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]: dispatch 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[67016]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/228788979' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1773772302"}]': finished 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[67016]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/342656354' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]: dispatch 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[67016]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]: dispatch 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[64470]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-02-10T07:41:51.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/228788979' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/1773772302"}]': finished 2024-02-10T07:41:51.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[64470]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T07:41:51.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/342656354' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]: dispatch 2024-02-10T07:41:51.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:51 smithi064 ceph-mon[64470]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]: dispatch 2024-02-10T07:41:52.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:52 smithi136 ceph-mon[57888]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]': finished 2024-02-10T07:41:52.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:52 smithi136 ceph-mon[57888]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T07:41:52.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:52 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1767590237' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/687938209"}]: dispatch 2024-02-10T07:41:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:52 smithi064 ceph-mon[64470]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]': finished 2024-02-10T07:41:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:52 smithi064 ceph-mon[64470]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T07:41:52.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:52 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1767590237' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/687938209"}]: dispatch 2024-02-10T07:41:52.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:52 smithi064 ceph-mon[67016]: from='client.? ' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/3596215613"}]': finished 2024-02-10T07:41:52.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:52 smithi064 ceph-mon[67016]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T07:41:52.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:52 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1767590237' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/687938209"}]: dispatch 2024-02-10T07:41:53.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:53 smithi136 ceph-mon[57888]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:41:53.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:53 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1767590237' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/687938209"}]': finished 2024-02-10T07:41:53.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:53 smithi136 ceph-mon[57888]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T07:41:53.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:53 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/437755878' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2991743476"}]: dispatch 2024-02-10T07:41:53.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[64470]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:41:53.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1767590237' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/687938209"}]': finished 2024-02-10T07:41:53.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[64470]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T07:41:53.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/437755878' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2991743476"}]: dispatch 2024-02-10T07:41:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[67016]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:41:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1767590237' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:0/687938209"}]': finished 2024-02-10T07:41:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[67016]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T07:41:53.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:53 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/437755878' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2991743476"}]: dispatch 2024-02-10T07:41:54.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:54 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/437755878' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2991743476"}]': finished 2024-02-10T07:41:54.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:54 smithi136 ceph-mon[57888]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T07:41:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:54 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/437755878' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2991743476"}]': finished 2024-02-10T07:41:54.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:54 smithi064 ceph-mon[64470]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T07:41:54.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:54 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/437755878' entity='client.iscsi.foo.smithi064.yotmxt' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.64:6800/2991743476"}]': finished 2024-02-10T07:41:54.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:54 smithi064 ceph-mon[67016]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T07:41:55.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:55 smithi136 ceph-mon[57888]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:55.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:55 smithi064 ceph-mon[64470]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:55.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:55 smithi064 ceph-mon[67016]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:41:57.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:57 smithi136 ceph-mon[57888]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-02-10T07:41:57.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:57 smithi064 ceph-mon[64470]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-02-10T07:41:57.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:57 smithi064 ceph-mon[67016]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-02-10T07:41:57.815 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:41:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:41:57] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-02-10T07:41:58.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:58 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:58.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:58 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:58.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:58 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:59 smithi136 ceph-mon[57888]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:41:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:41:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:59.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:59 smithi064 ceph-mon[64470]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:41:59.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:41:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:59.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:59 smithi064 ceph-mon[67016]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:41:59.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:41:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:01.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:01 smithi136 ceph-mon[57888]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:01.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:01 smithi064 ceph-mon[64470]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:01.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:01 smithi064 ceph-mon[67016]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:03.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:03 smithi136 ceph-mon[57888]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:42:03.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:03 smithi064 ceph-mon[64470]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:42:03.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:03 smithi064 ceph-mon[67016]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:42:05.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:05 smithi136 ceph-mon[57888]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-02-10T07:42:05.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:05 smithi064 ceph-mon[64470]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-02-10T07:42:05.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:05 smithi064 ceph-mon[67016]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-02-10T07:42:07.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:07 smithi136 ceph-mon[57888]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T07:42:07.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:07 smithi064 ceph-mon[64470]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T07:42:07.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:07 smithi064 ceph-mon[67016]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T07:42:07.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:42:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:42:07] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-02-10T07:42:08.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:08 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:08.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:08 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:08.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:08 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:09.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:09 smithi136 ceph-mon[57888]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:09.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:09 smithi064 ceph-mon[64470]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:09.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:09 smithi064 ceph-mon[67016]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:10.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:10 smithi136 ceph-mon[57888]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:10.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:10 smithi064 ceph-mon[64470]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:10.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:10 smithi064 ceph-mon[67016]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:13.472 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:13 smithi136 ceph-mon[57888]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T07:42:13.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:13 smithi064 ceph-mon[64470]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T07:42:13.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:13 smithi064 ceph-mon[67016]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T07:42:14.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:14.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:15.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:15 smithi064 ceph-mon[64470]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:15.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:15 smithi064 ceph-mon[67016]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:15.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:15 smithi136 ceph-mon[57888]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:17.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:17 smithi136 ceph-mon[57888]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:17.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:17 smithi064 ceph-mon[64470]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:17.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:17 smithi064 ceph-mon[67016]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:17.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:42:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:42:17] "GET /metrics HTTP/1.1" 200 38766 "" "Prometheus/2.33.4" 2024-02-10T07:42:18.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:18 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:18.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:18 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:18.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:18 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:19.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:19 smithi136 ceph-mon[57888]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:19.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:19 smithi064 ceph-mon[64470]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:19.816 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:19 smithi064 ceph-mon[67016]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:21.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:21 smithi136 ceph-mon[57888]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:21.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:21 smithi064 ceph-mon[64470]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:21.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:21 smithi064 ceph-mon[67016]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:23.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:23 smithi136 ceph-mon[57888]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:23.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:23 smithi064 ceph-mon[64470]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:23.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:23 smithi064 ceph-mon[67016]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:25.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:25 smithi136 ceph-mon[57888]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:25.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:25 smithi064 ceph-mon[64470]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:25.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:25 smithi064 ceph-mon[67016]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:27.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:27 smithi136 ceph-mon[57888]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:27.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:27 smithi064 ceph-mon[64470]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:27.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:27 smithi064 ceph-mon[67016]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:27.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:42:27 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:42:27] "GET /metrics HTTP/1.1" 200 38766 "" "Prometheus/2.33.4" 2024-02-10T07:42:28.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:28 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:28.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:28 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:28.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:28 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:29 smithi136 ceph-mon[57888]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:29.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:29 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:29.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:29 smithi064 ceph-mon[64470]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:29.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:29 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:29.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:29 smithi064 ceph-mon[67016]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:29.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:29 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:31.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:31 smithi136 ceph-mon[57888]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:31.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:31 smithi064 ceph-mon[64470]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:31.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:31 smithi064 ceph-mon[67016]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:33.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:33 smithi136 ceph-mon[57888]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:33.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:33 smithi064 ceph-mon[64470]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:33.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:33 smithi064 ceph-mon[67016]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:35.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:35 smithi136 ceph-mon[57888]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:35.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:35 smithi064 ceph-mon[64470]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:35.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:35 smithi064 ceph-mon[67016]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:37.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:37 smithi136 ceph-mon[57888]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:37.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:37 smithi064 ceph-mon[64470]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:37.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:37 smithi064 ceph-mon[67016]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:37.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:42:37 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:42:37] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-02-10T07:42:38.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:38 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:38.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:38 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:38.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:38 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:39.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:39 smithi136 ceph-mon[57888]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:39.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:39 smithi064 ceph-mon[64470]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:39.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:39 smithi064 ceph-mon[67016]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:41.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:41 smithi136 ceph-mon[57888]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:41.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:41 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:42:41.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:41 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:42:41.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:41 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:42:41.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[67016]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:41.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:42:41.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:42:41.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:42:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[64470]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:41.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:42:41.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:42:41.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:41 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' 2024-02-10T07:42:42.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:42 smithi136 ceph-mon[57888]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-02-10T07:42:42.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:42 smithi136 ceph-mon[57888]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:42.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:42 smithi064 ceph-mon[64470]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-02-10T07:42:42.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:42 smithi064 ceph-mon[64470]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:42.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:42 smithi064 ceph-mon[67016]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-02-10T07:42:42.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:42 smithi064 ceph-mon[67016]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:44 smithi136 ceph-mon[57888]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:44.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:44 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:44 smithi064 ceph-mon[64470]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:44.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:44 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:44.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:44 smithi064 ceph-mon[67016]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:44.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:44 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:46.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:46 smithi136 ceph-mon[57888]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:46.798 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:46 smithi064 ceph-mon[64470]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:46.798 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:46 smithi064 ceph-mon[67016]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:47.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:42:47 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:42:47] "GET /metrics HTTP/1.1" 200 38775 "" "Prometheus/2.33.4" 2024-02-10T07:42:48.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:48 smithi136 ceph-mon[57888]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-02-10T07:42:48.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:48 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:48.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:48 smithi064 ceph-mon[64470]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-02-10T07:42:48.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:48 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:48.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:48 smithi064 ceph-mon[67016]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-02-10T07:42:48.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:48 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:50.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:50 smithi136 ceph-mon[57888]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:50.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:50 smithi064 ceph-mon[64470]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:50.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:50 smithi064 ceph-mon[67016]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:42:51.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T07:42:52.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:52 smithi064 ceph-mon[64470]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:42:52.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:52 smithi064 ceph-mon[67016]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:42:52.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:52 smithi136 ceph-mon[57888]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:42:53.404 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 running (9m) 73s ago 14m - - 0.23.0 ba2b418f427c 3682b5fb1632 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:grafana.a smithi136 *:3000 running (8m) 74s ago 13m 54.8M - 8.3.5 dad864ee21e9 c2594a19899f 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.foo.smithi064.yotmxt smithi064 unknown 73s ago 13m - - 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi136 *:8443,9283,8765 running (8m) 74s ago 18m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 167aaac3e1e7 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:8443,9283,8765 running (8m) 73s ago 20m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 febd4e49df03 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (7m) 73s ago 20m - 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 874ccaeebd11 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi136 running (7m) 74s ago 18m 44.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c583d22115e3 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (7m) 73s ago 19m - 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 cc4764b36eb9 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.a smithi064 *:9100 running (9m) 73s ago 14m - - 1.3.1 1dbe0e931976 7490d8f51037 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter.b smithi136 *:9100 running (9m) 74s ago 14m 17.1M - 1.3.1 1dbe0e931976 45907070f49b 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (6m) 73s ago 18m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc41f5eaa554 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (5m) 73s ago 18m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 29a2c7716d71 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (5m) 73s ago 17m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 55fc618fb252 2024-02-10T07:42:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (4m) 73s ago 17m - 2736M 19.0.0-1361-g290212a1 c328ab9dcfc7 ece83d6c2afa 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi136 running (3m) 74s ago 16m 58.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 3fc5e5d0fa69 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi136 running (3m) 74s ago 16m 58.7M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 f694bfa437e7 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi136 running (2m) 74s ago 16m 47.4M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 59163ab2c153 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi136 running (2m) 74s ago 15m 53.1M 2992M 19.0.0-1361-g290212a1 c328ab9dcfc7 a997551fd755 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi136 *:9095 running (8m) 74s ago 14m 70.3M - 2.33.4 514e6a882f6e 82bac0a679e8 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi064.elxhce smithi064 *:8000 running (118s) 73s ago 13m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 78bc43a25dc5 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.smithi136.xkhtnu smithi136 *:8000 running (107s) 74s ago 13m 77.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 d0ccfe9ecfbe 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi064.dqyqnr smithi064 *:80 running (112s) 73s ago 13m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 dbbaac6e1a25 2024-02-10T07:42:53.406 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.smpl.smithi136.fharky smithi136 *:80 running (101s) 74s ago 13m 74.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 cd5364e8847f 2024-02-10T07:42:53.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T07:42:54.564 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:54 smithi064 ceph-mon[67016]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:54.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:54 smithi064 ceph-mon[64470]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:54.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:54 smithi136 ceph-mon[57888]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:42:55.417 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:55 smithi064 ceph-mon[64470]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:42:55.417 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:55 smithi064 ceph-mon[67016]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:42:55.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:55 smithi136 ceph-mon[57888]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:42:56.217 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:42:56.217 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": { 2024-02-10T07:42:56.217 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": { 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": { 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw": { 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "overall": { 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 17 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-10T07:42:56.218 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:42:56.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:56 smithi064 ceph-mon[64470]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:56.533 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:56 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1774736689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:42:56.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:56 smithi064 ceph-mon[67016]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:56.533 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:56 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1774736689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:42:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:56 smithi136 ceph-mon[57888]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:56.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:56 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1774736689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:42:56.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T07:42:57.814 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:42:57 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:42:57] "GET /metrics HTTP/1.1" 200 38775 "" "Prometheus/2.33.4" 2024-02-10T07:42:58.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:58 smithi136 ceph-mon[57888]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:58.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:58 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout: "in_progress": true, 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout: "services_complete": [ 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout: "mon", 2024-02-10T07:42:58.785 INFO:teuthology.orchestra.run.smithi064.stdout: "osd", 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw", 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr" 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout: "progress": "17/23 daemons upgraded", 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed.", 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout: "is_paused": true 2024-02-10T07:42:58.786 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:42:58.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:58 smithi064 ceph-mon[64470]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:58.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:58 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:58.786 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:58 smithi064 ceph-mon[67016]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:58.787 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:58 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:59.409 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:42:59 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:59.410 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:42:59 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:59.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T07:42:59.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:42:59 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:00 smithi064 ceph-mon[64470]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:00.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:00 smithi064 ceph-mon[64470]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:00.668 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:00 smithi064 ceph-mon[67016]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:00.669 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:00 smithi064 ceph-mon[67016]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:00.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:00 smithi136 ceph-mon[57888]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:00.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:00 smithi136 ceph-mon[57888]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:01.464 INFO:teuthology.orchestra.run.smithi064.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. 2024-02-10T07:43:01.464 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-10T07:43:01.464 INFO:teuthology.orchestra.run.smithi064.stdout: daemon iscsi.foo.smithi064.yotmxt on smithi064 is in unknown state 2024-02-10T07:43:01.464 INFO:teuthology.orchestra.run.smithi064.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi064.yotmxt on host smithi064 failed. 2024-02-10T07:43:01.464 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrade daemon: iscsi.foo.smithi064.yotmxt: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi064.yotmxt ... 2024-02-10T07:43:01.464 INFO:teuthology.orchestra.run.smithi064.stdout:Creating ceph-iscsi config... 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/iscsi-gateway.cfg 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:Write file: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/iscsi.foo.smithi064.yotmxt/tcmu-runner-entrypoint.sh 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d0ded8f14\x2dc7e5\x2d11ee\x2d95b8\x2d87774f69a715.slice/ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl start ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:Traceback (most recent call last): 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: return _run_code(code, main_globals, None, 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: exec(code, run_globals) 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5520, in 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 5508, in main 2024-02-10T07:43:01.465 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3038, in command_deploy_from 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3073, in _common_deploy 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 3093, in _deploy_daemon_container 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1029, in deploy_daemon 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/__main__.py", line 1165, in deploy_daemon_units 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout: File "/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/cephadm.7f1811529dedc2d427ce527b00a265922f87c3833bd58d6c53bc5cd3f45e5f56/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout:RuntimeError: Failed command: systemctl start ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt: Job for ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service failed because the control process exited with error code. 2024-02-10T07:43:01.466 INFO:teuthology.orchestra.run.smithi064.stdout:See "systemctl status ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" and "journalctl -xeu ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@iscsi.foo.smithi064.yotmxt.service" for details. 2024-02-10T07:43:02.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-10T07:43:02.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:02 smithi064 ceph-mon[64470]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:02.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:02 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/850553938' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:43:02.645 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:02 smithi064 ceph-mon[67016]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:02.645 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:02 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/850553938' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:43:02.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:02 smithi136 ceph-mon[57888]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:02.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:02 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/850553938' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:43:04.149 INFO:teuthology.orchestra.run.smithi064.stdout:true 2024-02-10T07:43:04.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:04 smithi064 ceph-mon[64470]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:04.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:04 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/1337160144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:43:04.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:04 smithi064 ceph-mon[67016]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:04.463 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:04 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/1337160144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:43:04.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:04 smithi136 ceph-mon[57888]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:04.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:04 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/1337160144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:43:04.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-10T07:43:06.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:06 smithi064 ceph-mon[64470]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:06.565 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:06 smithi064 ceph-mon[67016]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:06.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:06 smithi136 ceph-mon[57888]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:06.819 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T07:43:07.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-10T07:43:07.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:07 smithi064 ceph-mon[67016]: from='client.? 172.21.15.64:0/3626044776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:43:07.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:07 smithi064 ceph-mon[64470]: from='client.? 172.21.15.64:0/3626044776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:43:07.649 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:07 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:43:07] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-02-10T07:43:07.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:07 smithi136 ceph-mon[57888]: from='client.? 172.21.15.64:0/3626044776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:43:08.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:08 smithi064 ceph-mon[64470]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:08.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:08 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:08.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:08 smithi064 ceph-mon[67016]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:08.590 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:08 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:08.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:08 smithi136 ceph-mon[57888]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:08.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:08 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:09.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd 8 90s ago - 2024-02-10T07:43:10.065 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T07:43:10.075 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T07:43:10.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-02-10T07:43:10.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:10 smithi064 ceph-mon[64470]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:10.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:10 smithi064 ceph-mon[64470]: from='client.44814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:10.609 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:10 smithi064 ceph-mon[67016]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:10.609 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:10 smithi064 ceph-mon[67016]: from='client.44814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:10.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:10 smithi136 ceph-mon[57888]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:10.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:10 smithi136 ceph-mon[57888]: from='client.44814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:12.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:12 smithi136 ceph-mon[57888]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:12.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:12 smithi064 ceph-mon[64470]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:12.760 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:12 smithi064 ceph-mon[67016]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:12.760 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T07:43:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "image": "quay.io/ceph/ceph", 2024-02-10T07:43:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "registry": "quay.io", 2024-02-10T07:43:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "bare_image": "ceph/ceph", 2024-02-10T07:43:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "versions": [] 2024-02-10T07:43:12.760 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T07:43:13.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-02-10T07:43:13.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:13 smithi136 ceph-mon[57888]: from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:13.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:13 smithi064 ceph-mon[64470]: from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:13.813 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:13 smithi064 ceph-mon[67016]: from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:14.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:14 smithi136 ceph-mon[57888]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:14.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:14 smithi136 ceph-mon[57888]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:14.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:14 smithi064 ceph-mon[64470]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:14.784 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:14 smithi064 ceph-mon[64470]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:14.784 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:14 smithi064 ceph-mon[67016]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:14.785 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:14 smithi064 ceph-mon[67016]: from='mgr.44106 172.21.15.64:0/2365721020' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "16.2.0", 2024-02-10T07:43:16.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:16 smithi136 ceph-mon[57888]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:16.814 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:16 smithi064 ceph-mon[64470]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:16.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:16 smithi064 ceph-mon[67016]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:16.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-02-10T07:43:17.419 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:17 smithi064 ceph-mon[64470]: from='client.54772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:17.419 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:17 smithi064 ceph-mon[67016]: from='client.54772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:17.699 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:17 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: ::ffff:172.21.15.136 - - [10/Feb/2024:07:43:17] "GET /metrics HTTP/1.1" 200 38774 "" "Prometheus/2.33.4" 2024-02-10T07:43:17.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:17 smithi136 ceph-mon[57888]: from='client.54772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:18.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:18 smithi064 ceph-mon[64470]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:18.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:18 smithi064 ceph-mon[64470]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:18.596 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:18 smithi064 ceph-mon[67016]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:18.596 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:18 smithi064 ceph-mon[67016]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:18.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:18 smithi136 ceph-mon[57888]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:43:18.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:18 smithi136 ceph-mon[57888]: from='client.44727 -' entity='client.iscsi.foo.smithi064.yotmxt' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "v16.2.2", 2024-02-10T07:43:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "v16.2.2-20210505", 2024-02-10T07:43:19.939 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T07:43:19.949 INFO:tasks.cephadm:Teardown begin 2024-02-10T07:43:19.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:43:19.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:43:20.016 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T07:43:20.016 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:43:20.035 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:43:20.073 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T07:43:20.073 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T07:43:20.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a 2024-02-10T07:43:20.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 systemd[1]: Stopping Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:20.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:20 smithi136 ceph-mon[57888]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:20.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:20 smithi136 ceph-mon[57888]: from='client.54778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:20.761 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-mon[67016]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:20.761 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-mon[67016]: from='client.54778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:20.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-mon[64470]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:43:20.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-mon[64470]: from='client.54778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a[64466]: 2024-02-10T07:43:20.417+0000 7f4abfdeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a[64466]: 2024-02-10T07:43:20.417+0000 7f4abfdeb700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 podman[90666]: 2024-02-10 07:43:20.459616298 +0000 UTC m=+0.198672076 container died 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:43:20] ENGINE Bus STOPPING 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:43:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:43:20] ENGINE Bus STOPPED 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:43:20] ENGINE Bus STARTING 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: CherryPy Checker: 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: The Application mounted at '' has an empty config. 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:43:20.762 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:43:20] ENGINE Serving on http://:::9283 2024-02-10T07:43:20.763 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:20 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: [10/Feb/2024:07:43:20] ENGINE Bus STARTED 2024-02-10T07:43:21.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 podman[90666]: 2024-02-10 07:43:20.868479347 +0000 UTC m=+0.607535134 container cleanup 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T07:43:21.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:20 smithi064 bash[90666]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a 2024-02-10T07:43:21.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T07:43:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:21 smithi064 podman[90692]: 2024-02-10 07:43:21.053108269 +0000 UTC m=+0.586830190 container remove 874ccaeebd11da2755957b230b61b037c88549b3398869538c0088fb14399f9b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T07:43:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:21 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T07:43:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:21 smithi064 systemd[1]: Stopped Ceph mon.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:43:21.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 07:43:21 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.a.service: Consumed 8.631s CPU time. 2024-02-10T07:43:21.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:21.347 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T07:43:21.347 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-10T07:43:21.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c 2024-02-10T07:43:21.789 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:21 smithi064 systemd[1]: Stopping Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:22.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:21 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c[67012]: 2024-02-10T07:43:21.788+0000 7fa141e5a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:22.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:21 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c[67012]: 2024-02-10T07:43:21.788+0000 7fa141e5a700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-02-10T07:43:22.064 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 podman[90798]: 2024-02-10 07:43:22.034721909 +0000 UTC m=+0.465747470 container died cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-10T07:43:22.557 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 podman[90798]: 2024-02-10 07:43:22.381397801 +0000 UTC m=+0.812423342 container cleanup cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2) 2024-02-10T07:43:22.557 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 bash[90798]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c 2024-02-10T07:43:22.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T07:43:22.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 podman[90812]: 2024-02-10 07:43:22.557389739 +0000 UTC m=+0.517483830 container remove cc4764b36eb96caef4043847e7d0501fd24fe76cbdf3585a2692e1c1133fa4c9 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T07:43:22.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-10T07:43:22.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 systemd[1]: Stopped Ceph mon.c for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:43:22.814 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 07:43:22 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.c.service: Consumed 5.756s CPU time. 2024-02-10T07:43:22.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:22.846 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-10T07:43:22.846 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T07:43:22.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.b 2024-02-10T07:43:23.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:22 smithi136 systemd[1]: Stopping Ceph mon.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:23.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b[57884]: 2024-02-10T07:43:23.147+0000 7f6a661b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:23.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b[57884]: 2024-02-10T07:43:23.147+0000 7f6a661b2700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-02-10T07:43:23.473 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:23 smithi136 podman[76391]: 2024-02-10 07:43:23.301611095 +0000 UTC m=+0.256250595 container died c583d22115e375a89fe3c804e87a4e83bd8642fef39ec11ab9ad1a185be5a71c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T07:43:23.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:23 smithi136 podman[76391]: 2024-02-10 07:43:23.807234082 +0000 UTC m=+0.761873581 container cleanup c583d22115e375a89fe3c804e87a4e83bd8642fef39ec11ab9ad1a185be5a71c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-10T07:43:23.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Feb 10 07:43:23 smithi136 bash[76391]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mon-b 2024-02-10T07:43:24.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T07:43:24.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:24.277 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T07:43:24.277 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T07:43:24.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T07:43:24.564 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:24 smithi064 systemd[1]: Stopping Ceph mgr.y for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:24.927 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:24 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:43:24.601+0000 7f6e37b3b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:24.927 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:24 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y[60180]: 2024-02-10T07:43:24.601+0000 7f6e37b3b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T07:43:24.928 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:24 smithi064 podman[90918]: 2024-02-10 07:43:24.633342282 +0000 UTC m=+0.157176685 container died febd4e49df031318ce3519a4febb48e750c4c599871191fcbde03610c5dd5f50 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T07:43:25.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:25 smithi064 podman[90918]: 2024-02-10 07:43:25.12758211 +0000 UTC m=+0.651416493 container cleanup febd4e49df031318ce3519a4febb48e750c4c599871191fcbde03610c5dd5f50 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y, ceph=True, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T07:43:25.314 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 07:43:25 smithi064 bash[90918]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-y 2024-02-10T07:43:25.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T07:43:25.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:25.505 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T07:43:25.505 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T07:43:25.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x 2024-02-10T07:43:25.851 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:25 smithi136 systemd[1]: Stopping Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:26.138 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:25 smithi136 podman[76511]: 2024-02-10 07:43:25.85147637 +0000 UTC m=+0.142009954 container died 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T07:43:26.413 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 podman[76511]: 2024-02-10 07:43:26.212029963 +0000 UTC m=+0.502563551 container cleanup 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:43:26.413 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 bash[76511]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x 2024-02-10T07:43:26.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T07:43:26.666 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 podman[76526]: 2024-02-10 07:43:26.413629726 +0000 UTC m=+0.560476696 container remove 167aaac3e1e7c7d57f7364b53ba0a40c64cfb8a84f3bff654467b02bdc70f6bc (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T07:43:26.666 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:43:26.666 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-10T07:43:26.667 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 systemd[1]: Stopped Ceph mgr.x for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:43:26.667 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Feb 10 07:43:26 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@mgr.x.service: Consumed 20.081s CPU time. 2024-02-10T07:43:26.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:26.681 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T07:43:26.681 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T07:43:26.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.0 2024-02-10T07:43:27.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:26 smithi064 systemd[1]: Stopping Ceph osd.0 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:27.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:26 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:43:26.956+0000 7f26bef49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:27.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:26 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:43:26.956+0000 7f26bef49700 -1 osd.0 152 *** Got signal Terminated *** 2024-02-10T07:43:27.064 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:26 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0[70404]: 2024-02-10T07:43:26.956+0000 7f26bef49700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:43:32.564 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:32 smithi064 podman[91043]: 2024-02-10 07:43:32.301510425 +0000 UTC m=+5.434583367 container died fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:43:32.941 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:32 smithi064 podman[91043]: 2024-02-10 07:43:32.707027291 +0000 UTC m=+5.840100294 container cleanup fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T07:43:32.941 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:32 smithi064 bash[91043]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0 2024-02-10T07:43:33.233 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:32 smithi064 podman[91060]: 2024-02-10 07:43:32.941666328 +0000 UTC m=+0.636602276 container remove fc41f5eaa554356616752fde1d8124e42af63bbfd5c878dc5077d6481717e03b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T07:43:33.537 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:33 smithi064 podman[91150]: 2024-02-10 07:43:33.136733876 +0000 UTC m=+0.018428531 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:43:33.537 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:33 smithi064 podman[91150]: 2024-02-10 07:43:33.262121785 +0000 UTC m=+0.143816431 container create a9ce6d2b1b5356709c7fd0e59907434965f556ff79aecacdfc6969504f2c4994 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T07:43:33.537 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:33 smithi064 podman[91150]: 2024-02-10 07:43:33.462013869 +0000 UTC m=+0.343708517 container init a9ce6d2b1b5356709c7fd0e59907434965f556ff79aecacdfc6969504f2c4994 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T07:43:33.537 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:33 smithi064 podman[91150]: 2024-02-10 07:43:33.465371138 +0000 UTC m=+0.347065790 container start a9ce6d2b1b5356709c7fd0e59907434965f556ff79aecacdfc6969504f2c4994 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:43:33.537 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:33 smithi064 podman[91150]: 2024-02-10 07:43:33.53686578 +0000 UTC m=+0.418560432 container attach a9ce6d2b1b5356709c7fd0e59907434965f556ff79aecacdfc6969504f2c4994 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T07:43:33.814 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 07:43:33 smithi064 podman[91150]: 2024-02-10 07:43:33.761739385 +0000 UTC m=+0.643434032 container died a9ce6d2b1b5356709c7fd0e59907434965f556ff79aecacdfc6969504f2c4994 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T07:43:34.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:43:34.237Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:43:34.314 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:43:34.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:43:34.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T07:43:34.391 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:34.391 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T07:43:34.391 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T07:43:34.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.1 2024-02-10T07:43:34.599 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:34 smithi064 systemd[1]: Stopping Ceph osd.1 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:35.064 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:43:34.767+0000 7fb77f7da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:35.064 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:43:34.767+0000 7fb77f7da700 -1 osd.1 152 *** Got signal Terminated *** 2024-02-10T07:43:35.064 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:34 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1[72884]: 2024-02-10T07:43:34.767+0000 7fb77f7da700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:43:40.503 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:40 smithi064 podman[91255]: 2024-02-10 07:43:40.202918793 +0000 UTC m=+5.569794042 container died 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T07:43:40.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:40 smithi064 podman[91255]: 2024-02-10 07:43:40.618649355 +0000 UTC m=+5.985524603 container cleanup 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T07:43:40.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:40 smithi064 bash[91255]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1 2024-02-10T07:43:41.235 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:40 smithi064 podman[91289]: 2024-02-10 07:43:40.828184462 +0000 UTC m=+0.622343169 container remove 29a2c7716d7114c90e132635cc829390a0b88565aa62da6bd3fd1a688d31e778 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T07:43:41.548 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:41 smithi064 podman[91380]: 2024-02-10 07:43:41.138500382 +0000 UTC m=+0.017786671 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:43:41.548 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:41 smithi064 podman[91380]: 2024-02-10 07:43:41.237351546 +0000 UTC m=+0.116637828 container create b4982772af8449bf221d6c5dac6cddcdd3bc9d7d62722fa7bb6cfe52740f1ae2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131) 2024-02-10T07:43:41.548 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:41 smithi064 podman[91380]: 2024-02-10 07:43:41.448342697 +0000 UTC m=+0.327628981 container init b4982772af8449bf221d6c5dac6cddcdd3bc9d7d62722fa7bb6cfe52740f1ae2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T07:43:41.548 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:41 smithi064 podman[91380]: 2024-02-10 07:43:41.451337076 +0000 UTC m=+0.330623358 container start b4982772af8449bf221d6c5dac6cddcdd3bc9d7d62722fa7bb6cfe52740f1ae2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:43:41.548 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:41 smithi064 podman[91380]: 2024-02-10 07:43:41.548183917 +0000 UTC m=+0.427470211 container attach b4982772af8449bf221d6c5dac6cddcdd3bc9d7d62722fa7bb6cfe52740f1ae2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T07:43:41.814 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 07:43:41 smithi064 podman[91380]: 2024-02-10 07:43:41.746556312 +0000 UTC m=+0.625842600 container died b4982772af8449bf221d6c5dac6cddcdd3bc9d7d62722fa7bb6cfe52740f1ae2 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T07:43:42.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T07:43:42.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:42.438 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T07:43:42.438 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T07:43:42.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.2 2024-02-10T07:43:42.763 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:42 smithi064 systemd[1]: Stopping Ceph osd.2 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:43.064 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:42 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:43:42.762+0000 7f2ec9ebf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:43.065 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:42 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:43:42.762+0000 7f2ec9ebf700 -1 osd.2 152 *** Got signal Terminated *** 2024-02-10T07:43:43.065 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:42 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2[75359]: 2024-02-10T07:43:42.762+0000 7f2ec9ebf700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:43:43.472 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:43:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:43:43.069Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-02-10T07:43:43.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:43:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:43:43.072Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-02-10T07:43:43.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:43:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:43:43.072Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-02-10T07:43:43.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:43:43 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:43:43.072Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-02-10T07:43:44.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:43:44.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:43:44.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:43:44.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:43:44.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:44 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:43:44.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:43:48.440 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:48 smithi064 podman[91485]: 2024-02-10 07:43:48.10955312 +0000 UTC m=+5.438592468 container died 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-02-10T07:43:48.814 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:48 smithi064 podman[91485]: 2024-02-10 07:43:48.437723938 +0000 UTC m=+5.766763286 container cleanup 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T07:43:48.814 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:48 smithi064 bash[91485]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2 2024-02-10T07:43:48.814 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:48 smithi064 podman[91514]: 2024-02-10 07:43:48.639072244 +0000 UTC m=+0.526028609 container remove 55fc618fb2525457858a8ae0c452d209c243e590fe26b49657c279adbc1d462e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T07:43:49.309 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:49 smithi064 podman[91607]: 2024-02-10 07:43:48.960746159 +0000 UTC m=+0.017825915 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:43:49.309 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:49 smithi064 podman[91607]: 2024-02-10 07:43:49.106612429 +0000 UTC m=+0.163692224 container create 211b4dc3ef8b62fb9187522b684a6c2780dde4b20ab5c5071c0301ff2e802f5c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-10T07:43:49.564 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:49 smithi064 podman[91607]: 2024-02-10 07:43:49.309567684 +0000 UTC m=+0.366647430 container init 211b4dc3ef8b62fb9187522b684a6c2780dde4b20ab5c5071c0301ff2e802f5c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2) 2024-02-10T07:43:49.564 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:49 smithi064 podman[91607]: 2024-02-10 07:43:49.31292463 +0000 UTC m=+0.370004367 container start 211b4dc3ef8b62fb9187522b684a6c2780dde4b20ab5c5071c0301ff2e802f5c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True) 2024-02-10T07:43:49.564 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:49 smithi064 podman[91607]: 2024-02-10 07:43:49.392151367 +0000 UTC m=+0.449231126 container attach 211b4dc3ef8b62fb9187522b684a6c2780dde4b20ab5c5071c0301ff2e802f5c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-10T07:43:49.911 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 07:43:49 smithi064 podman[91607]: 2024-02-10 07:43:49.60991612 +0000 UTC m=+0.666995848 container died 211b4dc3ef8b62fb9187522b684a6c2780dde4b20ab5c5071c0301ff2e802f5c (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T07:43:50.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T07:43:50.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:50.231 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T07:43:50.231 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T07:43:50.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.3 2024-02-10T07:43:50.564 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:50 smithi064 systemd[1]: Stopping Ceph osd.3 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:51.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:50 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:43:50.589+0000 7f0bcaf61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:51.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:50 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:43:50.589+0000 7f0bcaf61700 -1 osd.3 152 *** Got signal Terminated *** 2024-02-10T07:43:51.064 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:50 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:43:50.589+0000 7f0bcaf61700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:43:53.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:53.161+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:53.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:53.552+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:53.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:53 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:53.687+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:43:54.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:54.185+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:54.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:54.199+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:43:54.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:54 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:43:54.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:43:54.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:54 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:43:54.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:43:54.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:43:54 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:43:54.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:43:54.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:54.536+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:54.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:54 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:54.657+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:43:55.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:55.135+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:55.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:55.181+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:43:55.814 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:55 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3[77819]: 2024-02-10T07:43:55.477+0000 7f0bc354d700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.495141+0000 front 2024-02-10T07:43:30.495308+0000 (oldest deadline 2024-02-10T07:43:55.195252+0000) 2024-02-10T07:43:55.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:55.582+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:55.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:55 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:55.664+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:43:56.243 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:55 smithi064 podman[91713]: 2024-02-10 07:43:55.92934592 +0000 UTC m=+5.471201269 container died ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-10T07:43:56.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:56 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:56.120+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:56.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:56 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:56.205+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:43:56.564 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:56 smithi064 podman[91713]: 2024-02-10 07:43:56.348089025 +0000 UTC m=+5.889944357 container cleanup ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:43:56.564 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:56 smithi064 bash[91713]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3 2024-02-10T07:43:56.908 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:56 smithi064 podman[91731]: 2024-02-10 07:43:56.582603512 +0000 UTC m=+0.644660835 container remove ece83d6c2afa1b15fe5025d36f5d5ec17aed6a88d2add3463906d015fb263710 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:43:56.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:56 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:56.621+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:56.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:56 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:56.696+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:43:57.227 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:56 smithi064 podman[91825]: 2024-02-10 07:43:56.908664554 +0000 UTC m=+0.109120083 container create 88e8e40dbf2d07fded5782213410c5f5e6af9d1c0ea276b646ec3c929c3c7644 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T07:43:57.227 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:56 smithi064 podman[91825]: 2024-02-10 07:43:56.817470857 +0000 UTC m=+0.017926398 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:43:57.227 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:57 smithi064 podman[91825]: 2024-02-10 07:43:57.152893919 +0000 UTC m=+0.353349447 container init 88e8e40dbf2d07fded5782213410c5f5e6af9d1c0ea276b646ec3c929c3c7644 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T07:43:57.227 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:57 smithi064 podman[91825]: 2024-02-10 07:43:57.155860306 +0000 UTC m=+0.356315835 container start 88e8e40dbf2d07fded5782213410c5f5e6af9d1c0ea276b646ec3c929c3c7644 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:43:57.228 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:57 smithi064 podman[91825]: 2024-02-10 07:43:57.227610581 +0000 UTC m=+0.428066109 container attach 88e8e40dbf2d07fded5782213410c5f5e6af9d1c0ea276b646ec3c929c3c7644 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:43:57.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:57.225+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:43:57.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:57.131+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:57.564 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 07:43:57 smithi064 podman[91825]: 2024-02-10 07:43:57.460062981 +0000 UTC m=+0.660518508 container died 88e8e40dbf2d07fded5782213410c5f5e6af9d1c0ea276b646ec3c929c3c7644 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T07:43:57.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:57.589+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:57.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:57 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:57.709+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:43:58.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T07:43:58.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:43:58.137 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T07:43:58.137 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T07:43:58.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.4 2024-02-10T07:43:58.448 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:58 smithi136 systemd[1]: Stopping Ceph osd.4 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:43:58.448 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:58.209+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:43:58.448 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:58.168+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:58.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:58.584+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:58.723 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:58.674+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:43:58.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:58.447+0000 7fa42dc25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:43:58.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:58.447+0000 7fa42dc25700 -1 osd.4 152 *** Got signal Terminated *** 2024-02-10T07:43:58.723 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:58 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:58.447+0000 7fa42dc25700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:43:59.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:43:59 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:43:59.192+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:43:59.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:43:59 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:43:59.173+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:43:59.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:43:59 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:43:59.612+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:43:59.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:43:59 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:43:59.717+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:00.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:00.140+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:00.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:00.228+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:44:00.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:00.228+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.727199+0000 front 2024-02-10T07:43:36.727122+0000 (oldest deadline 2024-02-10T07:44:00.227041+0000) 2024-02-10T07:44:00.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:00.624+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:00.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:00 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:00.702+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:01.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:01.233+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:44:01.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:01.233+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.727199+0000 front 2024-02-10T07:43:36.727122+0000 (oldest deadline 2024-02-10T07:44:00.227041+0000) 2024-02-10T07:44:01.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:01.091+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:01.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:01.091+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:01.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:01.593+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:01.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:01 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:01.696+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:02.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:02.206+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:44:02.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:02.206+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.727199+0000 front 2024-02-10T07:43:36.727122+0000 (oldest deadline 2024-02-10T07:44:00.227041+0000) 2024-02-10T07:44:02.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:02.046+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:02.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:02.047+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:02.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:02.558+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:02.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:02.741+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:02.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:02 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:02.741+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:03.472 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:03.176+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.126093+0000 front 2024-02-10T07:43:29.126231+0000 (oldest deadline 2024-02-10T07:43:53.826133+0000) 2024-02-10T07:44:03.473 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4[61155]: 2024-02-10T07:44:03.176+0000 7fa42a219700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.727199+0000 front 2024-02-10T07:43:36.727122+0000 (oldest deadline 2024-02-10T07:44:00.227041+0000) 2024-02-10T07:44:03.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:03.006+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:03.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:03.006+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:03.813 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:03.771+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:03.813 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:03.771+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:03.813 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:03.549+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:03.813 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:03 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:03.549+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:03.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:03.545Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:03.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:03 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:03.546Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:04.091 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:03 smithi136 podman[76645]: 2024-02-10 07:44:03.813360524 +0000 UTC m=+5.481218805 container died 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T07:44:04.092 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:04.035+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:04.092 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:04.035+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:04.415 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:04 smithi136 podman[76645]: 2024-02-10 07:44:04.231001051 +0000 UTC m=+5.898859334 container cleanup 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:44:04.416 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:04 smithi136 bash[76645]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4 2024-02-10T07:44:04.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:44:04.239Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:04.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:04.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:04.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:04 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:04.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:04.722 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:04 smithi136 podman[76662]: 2024-02-10 07:44:04.415624041 +0000 UTC m=+0.598262170 container remove 3fc5e5d0fa69ab39028bc15c53e0a84f07108db97d09a3b5aa505c3ca95d879a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T07:44:04.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:04.569+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:04.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:04.569+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:05.014 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:04.760+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:05.014 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:04 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:04.760+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:05.014 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:04 smithi136 podman[76757]: 2024-02-10 07:44:04.760365914 +0000 UTC m=+0.017401622 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:44:05.014 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:04 smithi136 podman[76757]: 2024-02-10 07:44:04.85812311 +0000 UTC m=+0.115158810 container create 66bb3b7629098f102032d3c302521ba3a4729d144cffbba5711e0207e444fa7b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T07:44:05.394 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:05 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:05.013+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:05.394 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:05 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:05.013+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:05.395 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:05 smithi136 podman[76757]: 2024-02-10 07:44:05.094681261 +0000 UTC m=+0.351716964 container init 66bb3b7629098f102032d3c302521ba3a4729d144cffbba5711e0207e444fa7b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:44:05.395 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:05 smithi136 podman[76757]: 2024-02-10 07:44:05.097939438 +0000 UTC m=+0.354975141 container start 66bb3b7629098f102032d3c302521ba3a4729d144cffbba5711e0207e444fa7b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:44:05.395 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:05 smithi136 podman[76757]: 2024-02-10 07:44:05.169430406 +0000 UTC m=+0.426466110 container attach 66bb3b7629098f102032d3c302521ba3a4729d144cffbba5711e0207e444fa7b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:44:05.395 INFO:journalctl@ceph.osd.4.smithi136.stdout:Feb 10 07:44:05 smithi136 podman[76757]: 2024-02-10 07:44:05.394654755 +0000 UTC m=+0.651690461 container died 66bb3b7629098f102032d3c302521ba3a4729d144cffbba5711e0207e444fa7b (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-10T07:44:05.690 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:05 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:05.548+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:05.690 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:05 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:05.548+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:05.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:05 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:05.797+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:05.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:05 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:05.797+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:06.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T07:44:06.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:44:06.042 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T07:44:06.042 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T07:44:06.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.5 2024-02-10T07:44:06.418 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:06 smithi136 systemd[1]: Stopping Ceph osd.5 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:06.418 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:06.054+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:06.418 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:06.054+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:06.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:06.417+0000 7f92be2cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:44:06.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:06.417+0000 7f92be2cb700 -1 osd.5 152 *** Got signal Terminated *** 2024-02-10T07:44:06.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:06.417+0000 7f92be2cb700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:44:06.723 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:06.547+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:06.723 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:06.547+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:07.222 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:06.823+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:07.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:06 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:06.823+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:07.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:07.021+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:07.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:07.021+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:07.854 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:07.597+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:07.855 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:07.597+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:08.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:08.001+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:08.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:08.001+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:08.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:07.854+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:08.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:07 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:07.854+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:08.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:08.627+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:08.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:08.627+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:08.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:08.813+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:08.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:08.813+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:09.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:08.992+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:09.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:08 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:08.992+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:09.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:09.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:09.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:09.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:09.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:09.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:09.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:09.649+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:09.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:09.649+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:09.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:09 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:09.649+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:42.887556+0000 front 2024-02-10T07:43:42.887667+0000 (oldest deadline 2024-02-10T07:44:08.787693+0000) 2024-02-10T07:44:10.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:10.025+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:10.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:10.025+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:10.972 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:10.612+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:30.586251+0000 front 2024-02-10T07:43:30.586405+0000 (oldest deadline 2024-02-10T07:43:52.886180+0000) 2024-02-10T07:44:10.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:10.612+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:36.987277+0000 front 2024-02-10T07:43:36.987206+0000 (oldest deadline 2024-02-10T07:44:02.887096+0000) 2024-02-10T07:44:10.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5[63475]: 2024-02-10T07:44:10.612+0000 7f92b68b7700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:42.887556+0000 front 2024-02-10T07:43:42.887667+0000 (oldest deadline 2024-02-10T07:44:08.787693+0000) 2024-02-10T07:44:10.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:10.766+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:10.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:10.766+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:10.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:10.766+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:11.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:11.039+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:11.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:11.039+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:11.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:11.039+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:12.060 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:11 smithi136 podman[76873]: 2024-02-10 07:44:11.916113083 +0000 UTC m=+5.635161592 container died f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-10T07:44:12.060 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:11.783+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:12.060 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:11.783+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:12.060 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:11 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:11.783+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:12.315 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:12 smithi136 podman[76873]: 2024-02-10 07:44:12.300710014 +0000 UTC m=+6.019758519 container cleanup f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-10T07:44:12.315 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:12 smithi136 bash[76873]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5 2024-02-10T07:44:12.315 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:12.059+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:12.316 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:12.059+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:12.316 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:12.059+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:12.822 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:12 smithi136 podman[76890]: 2024-02-10 07:44:12.568737781 +0000 UTC m=+0.649556501 container remove f694bfa437e72531b4af2d861f90fbbd040ce1be62152aecf1be1e1694705cfa (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T07:44:13.085 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:12.822+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:13.086 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:12.822+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:13.086 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:12 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:12.822+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:13.086 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:12 smithi136 podman[76980]: 2024-02-10 07:44:12.911266748 +0000 UTC m=+0.111531284 container create 30088c9da164069b33a0a90e8fe4240e11aa7d7417842089bcf773e5f0c56417 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2) 2024-02-10T07:44:13.086 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:12 smithi136 podman[76980]: 2024-02-10 07:44:12.817424233 +0000 UTC m=+0.017688734 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:44:13.401 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:13.085+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:13.402 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:13.085+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:13.402 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:13.085+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:13.402 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:13 smithi136 podman[76980]: 2024-02-10 07:44:13.107114833 +0000 UTC m=+0.307379333 container init 30088c9da164069b33a0a90e8fe4240e11aa7d7417842089bcf773e5f0c56417 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:44:13.402 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:13 smithi136 podman[76980]: 2024-02-10 07:44:13.110703341 +0000 UTC m=+0.310967837 container start 30088c9da164069b33a0a90e8fe4240e11aa7d7417842089bcf773e5f0c56417 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T07:44:13.402 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:13 smithi136 podman[76980]: 2024-02-10 07:44:13.180227034 +0000 UTC m=+0.380491543 container attach 30088c9da164069b33a0a90e8fe4240e11aa7d7417842089bcf773e5f0c56417 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:44:13.722 INFO:journalctl@ceph.osd.5.smithi136.stdout:Feb 10 07:44:13 smithi136 podman[76980]: 2024-02-10 07:44:13.402409381 +0000 UTC m=+0.602673882 container died 30088c9da164069b33a0a90e8fe4240e11aa7d7417842089bcf773e5f0c56417 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T07:44:13.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:44:13.546Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:13.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:13.547Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:13.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:13 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:13.547Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:14.008 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:13.785+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:14.008 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:13.785+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:14.008 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:13 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:13.785+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:14.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T07:44:14.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:44:14.075 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T07:44:14.075 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T07:44:14.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.6 2024-02-10T07:44:14.387 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:14.079+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:14.387 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:14.079+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:14.387 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:14.079+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:14.387 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 systemd[1]: Stopping Ceph osd.6 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:14.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:44:14.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:14.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:14.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:14.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:14 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:14.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:14.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:14.386+0000 7fc411025700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:44:14.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:14.386+0000 7fc411025700 -1 osd.6 152 *** Got signal Terminated *** 2024-02-10T07:44:14.722 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:14.386+0000 7fc411025700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:44:15.222 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:15.040+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:15.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:15.040+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:15.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:15.040+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:15.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:14.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:15.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:14.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:15.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:14 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:14.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:15.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:15.781+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:15.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:15.781+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:15.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:15 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:15.781+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:16.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:16.014+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:16.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:16.014+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:16.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:16.014+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:17.222 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:16.767+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:17.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:16.767+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:17.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:16.767+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:17.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:16.965+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:17.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:16.965+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:17.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:16 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:16.965+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:18.222 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:17.953+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:18.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:17.953+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:18.223 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:17.953+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:18.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:17.725+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:18.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:17.725+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:18.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:17.725+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:18.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:17 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:17.725+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:18.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:18.682+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:18.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:18.682+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:18.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:18.682+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:18.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:18.682+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:18.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:18.682+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:19.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:18.975+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:28.934787+0000 front 2024-02-10T07:43:28.934731+0000 (oldest deadline 2024-02-10T07:43:53.034540+0000) 2024-02-10T07:44:19.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:18.975+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.935511+0000 front 2024-02-10T07:43:38.935410+0000 (oldest deadline 2024-02-10T07:44:00.635347+0000) 2024-02-10T07:44:19.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:18.975+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:44.136399+0000 front 2024-02-10T07:43:44.136359+0000 (oldest deadline 2024-02-10T07:44:10.036233+0000) 2024-02-10T07:44:19.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:18 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6[65824]: 2024-02-10T07:44:18.975+0000 7fc40d619700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:52.937337+0000 front 2024-02-10T07:43:52.937282+0000 (oldest deadline 2024-02-10T07:44:18.137566+0000) 2024-02-10T07:44:19.972 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:19 smithi136 podman[77085]: 2024-02-10 07:44:19.804596416 +0000 UTC m=+5.510599394 container died 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:44:19.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:19 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:19.688+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:19.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:19 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:19.688+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:19.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:19 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:19.688+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:19.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:19 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:19.688+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:19.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:19 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:19.688+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:20.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:20 smithi136 podman[77085]: 2024-02-10 07:44:20.261171335 +0000 UTC m=+5.967174312 container cleanup 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:44:20.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:20 smithi136 bash[77085]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6 2024-02-10T07:44:20.875 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:20 smithi136 podman[77102]: 2024-02-10 07:44:20.520985809 +0000 UTC m=+0.713591005 container remove 59163ab2c1530deba34a64ab0086262adea7d9e961e528b0f49071bfb919714a (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T07:44:20.876 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:20 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:20.706+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:20.876 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:20 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:20.706+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:20.876 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:20 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:20.706+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:20.876 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:20 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:20.706+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:20.876 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:20 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:20.706+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:21.137 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:20 smithi136 podman[77195]: 2024-02-10 07:44:20.779312371 +0000 UTC m=+0.018135460 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:44:21.137 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:20 smithi136 podman[77195]: 2024-02-10 07:44:20.880115813 +0000 UTC m=+0.118938902 container create 5163d2902798bbbf57ffd0fc605897d5c9b748af3544e681957274a8d3d66599 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T07:44:21.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:21 smithi136 podman[77195]: 2024-02-10 07:44:21.24959219 +0000 UTC m=+0.488415279 container init 5163d2902798bbbf57ffd0fc605897d5c9b748af3544e681957274a8d3d66599 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T07:44:21.472 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:21 smithi136 podman[77195]: 2024-02-10 07:44:21.252643118 +0000 UTC m=+0.491466210 container start 5163d2902798bbbf57ffd0fc605897d5c9b748af3544e681957274a8d3d66599 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:44:21.473 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:21 smithi136 podman[77195]: 2024-02-10 07:44:21.324390698 +0000 UTC m=+0.563213810 container attach 5163d2902798bbbf57ffd0fc605897d5c9b748af3544e681957274a8d3d66599 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-10T07:44:21.972 INFO:journalctl@ceph.osd.6.smithi136.stdout:Feb 10 07:44:21 smithi136 podman[77195]: 2024-02-10 07:44:21.540440651 +0000 UTC m=+0.779263740 container died 5163d2902798bbbf57ffd0fc605897d5c9b748af3544e681957274a8d3d66599 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-10T07:44:21.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:21.739+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:21.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:21.740+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:21.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:21.740+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:21.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:21.740+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:21.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:21 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:21.740+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:22.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T07:44:22.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:44:22.079 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T07:44:22.079 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T07:44:22.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.7 2024-02-10T07:44:22.389 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 systemd[1]: Stopping Ceph osd.7 for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:22.389 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.388+0000 7f9f816bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:44:22.389 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.388+0000 7f9f816bc700 -1 osd.7 152 *** Got signal Terminated *** 2024-02-10T07:44:22.389 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.388+0000 7f9f816bc700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:44:23.222 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.731+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:23.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.731+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:23.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.731+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:23.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.731+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:23.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:22 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:22.731+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:44:23.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:23.548Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:23.814 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:23 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:23.548Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:24.124 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:23.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:24.125 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:23.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:24.125 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:23.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:24.125 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:23.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:24.125 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:23 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:23.764+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:24.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:24 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=error ts=2024-02-10T07:44:24.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:24.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:24 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi064.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.64:8443: connect: connection refused" 2024-02-10T07:44:24.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:24 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=warn ts=2024-02-10T07:44:24.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi136.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.136:8443: connect: connection refused" 2024-02-10T07:44:24.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:24 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:24.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:24.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:24 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:24.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:24.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:24 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:24.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:24.972 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:24 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:24.763+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:24.973 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:24 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:24.763+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:26.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:25 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:25.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:26.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:25 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:25.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:26.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:25 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:25.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:26.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:25 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:25.806+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:26.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:25 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:25.806+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:27.222 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:26 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:26.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T07:43:29.066691+0000 front 2024-02-10T07:43:29.066820+0000 (oldest deadline 2024-02-10T07:43:53.166550+0000) 2024-02-10T07:44:27.222 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:26 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:26.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T07:43:38.968175+0000 front 2024-02-10T07:43:38.968069+0000 (oldest deadline 2024-02-10T07:44:01.868093+0000) 2024-02-10T07:44:27.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:26 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:26.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T07:43:45.269622+0000 front 2024-02-10T07:43:45.269525+0000 (oldest deadline 2024-02-10T07:44:09.369504+0000) 2024-02-10T07:44:27.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:26 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:26.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T07:43:55.171076+0000 front 2024-02-10T07:43:55.171029+0000 (oldest deadline 2024-02-10T07:44:18.670995+0000) 2024-02-10T07:44:27.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:26 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:26.808+0000 7f9f79ca8700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-02-10T07:44:02.171967+0000 front 2024-02-10T07:44:02.172087+0000 (oldest deadline 2024-02-10T07:44:26.272001+0000) 2024-02-10T07:44:27.223 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:26 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7[68157]: 2024-02-10T07:44:26.808+0000 7f9f79ca8700 -1 osd.7 152 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.64499.0:32 2.5 2:a475fea6:::realms.16dd835a-3272-4f17-99fb-3d022c71fae2.control:head [watch ping cookie 94464253056640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-02-10T07:44:28.057 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:27 smithi136 podman[77300]: 2024-02-10 07:44:27.751798743 +0000 UTC m=+5.453908685 container died a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T07:44:28.456 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:28 smithi136 podman[77300]: 2024-02-10 07:44:28.246738312 +0000 UTC m=+5.948848303 container cleanup a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-10T07:44:28.456 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:28 smithi136 bash[77300]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7 2024-02-10T07:44:28.722 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:28 smithi136 podman[77329]: 2024-02-10 07:44:28.456544534 +0000 UTC m=+0.696358839 container remove a997551fd75541d3717573ce52a2beda8f6a58b377a7d2c53164ce1f04171177 (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-10T07:44:29.167 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:28 smithi136 podman[77418]: 2024-02-10 07:44:28.774509285 +0000 UTC m=+0.019420314 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:44:29.167 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:28 smithi136 podman[77418]: 2024-02-10 07:44:28.882265718 +0000 UTC m=+0.127176747 container create fe1d867587b786f0cbd375345521574e3fb18a5a9c208024512eb8945bd0fc0e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-02-10T07:44:29.168 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:29 smithi136 podman[77418]: 2024-02-10 07:44:29.101428092 +0000 UTC m=+0.346339120 container init fe1d867587b786f0cbd375345521574e3fb18a5a9c208024512eb8945bd0fc0e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:44:29.168 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:29 smithi136 podman[77418]: 2024-02-10 07:44:29.104530518 +0000 UTC m=+0.349441547 container start fe1d867587b786f0cbd375345521574e3fb18a5a9c208024512eb8945bd0fc0e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:44:29.168 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:29 smithi136 podman[77418]: 2024-02-10 07:44:29.167843581 +0000 UTC m=+0.412754632 container attach fe1d867587b786f0cbd375345521574e3fb18a5a9c208024512eb8945bd0fc0e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T07:44:29.472 INFO:journalctl@ceph.osd.7.smithi136.stdout:Feb 10 07:44:29 smithi136 podman[77418]: 2024-02-10 07:44:29.396109394 +0000 UTC m=+0.641020422 container died fe1d867587b786f0cbd375345521574e3fb18a5a9c208024512eb8945bd0fc0e (image=quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:44:29.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T07:44:29.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:44:29.990 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T07:44:29.990 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T07:44:29.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T07:44:30.198 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 systemd[1]: Stopping Ceph prometheus.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:30.472 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.198Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.198Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.198Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.198Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.198Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.198Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.199Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.199Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.199Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.199Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.199Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a[53781]: ts=2024-02-10T07:44:30.199Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T07:44:30.473 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 podman[77498]: 2024-02-10 07:44:30.209379253 +0000 UTC m=+0.120677287 container died 82bac0a679e8e2782628b270a6c311f2bbad1ea677c733ed01b448ec3711126c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:30.832 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 podman[77498]: 2024-02-10 07:44:30.590378919 +0000 UTC m=+0.501676966 container cleanup 82bac0a679e8e2782628b270a6c311f2bbad1ea677c733ed01b448ec3711126c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:30.832 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 bash[77498]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T07:44:30.833 INFO:journalctl@ceph.prometheus.a.smithi136.stdout:Feb 10 07:44:30 smithi136 bash[77525]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T07:44:30.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T07:44:30.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:44:30.875 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T07:44:30.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T07:44:32.733 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:32 smithi064 systemd[1]: Stopping Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:32.733 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:32 smithi064 podman[92192]: 2024-02-10 07:44:32.49208045 +0000 UTC m=+0.107922201 container died 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:32.998 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:32 smithi064 podman[92192]: 2024-02-10 07:44:32.831485435 +0000 UTC m=+0.447327193 container cleanup 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:32.998 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:32 smithi064 bash[92192]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T07:44:33.249 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:32 smithi064 podman[92207]: 2024-02-10 07:44:32.999016885 +0000 UTC m=+0.507801088 container remove 7490d8f51037d02adc6c033ad64db6dbcf2fe1d9f464441bf918c863f5141fd6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:33.249 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:33 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:44:33.249 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:33 smithi064 bash[92219]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T07:44:33.249 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:33 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T07:44:33.249 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:33 smithi064 systemd[1]: Stopped Ceph node-exporter.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:44:33.249 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 07:44:33 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 3.865s CPU time. 2024-02-10T07:44:33.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:33 smithi064 systemd[1]: Stopping Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:33.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:33 smithi064 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a[58932]: level=info ts=2024-02-10T07:44:33.360Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:44:33.564 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:33 smithi064 podman[92271]: 2024-02-10 07:44:33.401641593 +0000 UTC m=+0.144323193 container died 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:34.028 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:33 smithi064 podman[92271]: 2024-02-10 07:44:33.76060704 +0000 UTC m=+0.503288639 container cleanup 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:34.028 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:33 smithi064 bash[92271]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T07:44:34.028 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:33 smithi064 bash[92298]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T07:44:34.283 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:34 smithi064 podman[92285]: 2024-02-10 07:44:34.028028234 +0000 UTC m=+0.616811527 container remove 3682b5fb163286eadc870ddfc1e9f03011be530199197a5c65a46db7029a18cc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:44:34.283 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:34 smithi064 podman[92285]: 2024-02-10 07:44:34.095243162 +0000 UTC m=+0.684026456 volume remove fa5fdc8c9c9c7342b8b1a8fc4cffa9081d1aa283a91b99e98f5b9af70ebc567e 2024-02-10T07:44:34.283 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:34 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T07:44:34.283 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:34 smithi064 systemd[1]: Stopped Ceph alertmanager.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:44:34.283 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 07:44:34 smithi064 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.025s CPU time. 2024-02-10T07:45:08.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T07:45:10.037 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:09 smithi136 systemd[1]: Stopping Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:45:10.037 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:09 smithi136 podman[77858]: 2024-02-10 07:45:09.873422004 +0000 UTC m=+0.119527653 container died 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:45:10.560 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 podman[77858]: 2024-02-10 07:45:10.296682204 +0000 UTC m=+0.542787862 container cleanup 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 bash[77858]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 bash[77886]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 podman[77872]: 2024-02-10 07:45:10.539871009 +0000 UTC m=+0.668853733 container remove 45907070f49b753bc714a7a8cdce0cb1f48b5701a839167fa6b6775f531baca7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 systemd[1]: Stopped Ceph node-exporter.b for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:45:10.561 INFO:journalctl@ceph.node-exporter.b.smithi136.stdout:Feb 10 07:45:10 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 3.746s CPU time. 2024-02-10T07:45:10.851 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:10 smithi136 systemd[1]: Stopping Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:45:11.158 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:10 smithi136 ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a[50910]: t=2024-02-10T07:45:10+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T07:45:11.158 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:10 smithi136 podman[77947]: 2024-02-10 07:45:10.862491023 +0000 UTC m=+0.125421101 container died c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, io.openshift.expose-services=, distribution-scope=public, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, name=ubi8, build-date=2022-03-28T10:36:18.413762) 2024-02-10T07:45:11.159 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 podman[77947]: 2024-02-10 07:45:11.159076359 +0000 UTC m=+0.422006445 container cleanup c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, release=236.1648460182, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8) 2024-02-10T07:45:11.472 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 bash[77947]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T07:45:11.472 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 bash[77973]: Error: no container with name or ID "ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T07:45:11.473 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 podman[77961]: 2024-02-10 07:45:11.385256641 +0000 UTC m=+0.525825607 container remove c2594a19899fb77bef9015787367266e7b63b5c00232ba34edae364d2d3953ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715-grafana-a, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.openshift.expose-services=, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762) 2024-02-10T07:45:11.473 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-10T07:45:11.473 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 systemd[1]: Stopped Ceph grafana.a for 0ded8f14-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:45:11.473 INFO:journalctl@ceph.grafana.a.smithi136.stdout:Feb 10 07:45:11 smithi136 systemd[1]: ceph-0ded8f14-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Consumed 4.054s CPU time. 2024-02-10T07:45:34.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:45:34.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:45:34.564 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T07:45:34.567 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/remote/smithi064/crash 2024-02-10T07:45:34.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:45:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T07:45:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-02-10T07:45:34.612 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/remote/smithi136/crash 2024-02-10T07:45:34.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:45:34.656 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T07:45:34.657 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-02-10T07:45:34.658 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T07:45:34.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:45:34.689 INFO:teuthology.orchestra.run.smithi064.stdout:2024-02-10T07:26:33.709877+0000 mon.a (mon.0) 466 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:45:34.689 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T07:45:34.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[SEC\]' /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:45:34.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]' /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:45:34.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[WRN\]' /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:45:34.901 INFO:teuthology.orchestra.run.smithi064.stdout:2024-02-10T07:26:33.709877+0000 mon.a (mon.0) 466 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:45:34.901 INFO:tasks.cephadm:Compressing logs... 2024-02-10T07:45:34.902 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:45:34.944 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:45:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:45:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-02-10T07:45:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T07:45:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi064.elxhce.log 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi064.elxhce.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.smpl.smithi064.dqyqnr.log 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr: 76.7% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi064.elxhce.log.gz 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.smpl.smithi064.dqyqnr.log: /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr: 76.7% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.smpl.smithi064.dqyqnr.log.gz 2024-02-10T07:45:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/tcmu-runner.log 2024-02-10T07:45:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.3.log: /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T07:45:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/tcmu-runner.log: 84.6% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/tcmu-runner.log.gz 2024-02-10T07:45:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:45:34.973 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:45:34.973 INFO:teuthology.orchestra.run.smithi136.stderr:: No such file or directory 2024-02-10T07:45:34.973 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T07:45:34.973 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.smpl.smithi136.fharky.log 2024-02-10T07:45:34.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:45:34.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.smpl.smithi136.fharky.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T07:45:34.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T07:45:34.975 INFO:teuthology.orchestra.run.smithi136.stderr: 76.8% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.smpl.smithi136.fharky.log.gz 2024-02-10T07:45:34.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T07:45:34.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T07:45:34.976 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.0.log: 94.2% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:45:34.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.5.log: 90.5% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:45:34.979 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:45:34.979 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:45:34.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mgr.x.log: 88.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:45:34.984 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:45:34.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:45:34.985 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:45:34.987 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:45:34.987 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T07:45:34.988 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log: 90.7% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:45:34.989 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:45:34.990 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log: 87.8% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:45:34.990 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T07:45:34.992 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.2.log: 93.1% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:45:34.992 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T07:45:35.003 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi136.xkhtnu.log 2024-02-10T07:45:35.003 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.b.log: 91.4% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:45:35.004 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi136.xkhtnu.log: 76.8% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.smithi136.xkhtnu.log.gz 2024-02-10T07:45:35.019 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:45:35.029 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T07:45:35.333 INFO:teuthology.orchestra.run.smithi064.stderr: 88.6% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T07:45:35.420 INFO:teuthology.orchestra.run.smithi136.stderr: 92.4% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T07:45:35.469 INFO:teuthology.orchestra.run.smithi064.stderr: 92.1% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T07:45:35.984 INFO:teuthology.orchestra.run.smithi064.stderr: 91.0% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T07:45:36.159 INFO:teuthology.orchestra.run.smithi136.stderr: 93.4% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T07:45:36.162 INFO:teuthology.orchestra.run.smithi064.stderr: 93.5% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T07:45:36.398 INFO:teuthology.orchestra.run.smithi136.stderr: 93.5% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T07:45:36.405 INFO:teuthology.orchestra.run.smithi136.stderr: 93.6% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T07:45:36.598 INFO:teuthology.orchestra.run.smithi064.stderr: 93.7% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T07:45:36.623 INFO:teuthology.orchestra.run.smithi136.stderr: 93.7% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T07:45:36.625 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-02-10T07:45:36.625 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m1.667s 2024-02-10T07:45:36.626 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m5.979s 2024-02-10T07:45:36.626 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.231s 2024-02-10T07:45:36.637 INFO:teuthology.orchestra.run.smithi064.stderr: 93.7% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T07:45:36.854 INFO:teuthology.orchestra.run.smithi064.stderr: 93.6% -- replaced with /var/log/ceph/0ded8f14-c7e5-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T07:45:36.856 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T07:45:36.856 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m1.901s 2024-02-10T07:45:36.857 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m7.985s 2024-02-10T07:45:36.857 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.256s 2024-02-10T07:45:36.857 INFO:tasks.cephadm:Archiving logs... 2024-02-10T07:45:36.857 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/remote/smithi064/log 2024-02-10T07:45:36.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T07:45:37.854 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/remote/smithi136/log 2024-02-10T07:45:37.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T07:45:38.682 INFO:tasks.cephadm:Removing cluster... 2024-02-10T07:45:38.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 --force 2024-02-10T07:45:38.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ded8f14-c7e5-11ee-95b8-87774f69a715 --force 2024-02-10T07:45:39.262 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T07:45:39.262 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T07:45:39.278 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T07:45:39.292 INFO:tasks.cephadm:Teardown complete 2024-02-10T07:45:39.292 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T07:45:39.302 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T07:45:39.302 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:45:39.321 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:45:39.332 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:45:39.335 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:45:39.335 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-02-10T07:45:39.336 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +653us[ +656us] +/- 87ms 2024-02-10T07:45:39.336 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -359us[ -357us] +/- 55ms 2024-02-10T07:45:39.336 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -89us[ -89us] +/- 61ms 2024-02-10T07:45:39.336 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:45:39.345 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:45:39.348 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:45:39.349 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-02-10T07:45:39.349 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 +650us[ +675us] +/- 88ms 2024-02-10T07:45:39.349 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -337us[ -313us] +/- 56ms 2024-02-10T07:45:39.349 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -93us[ -69us] +/- 61ms 2024-02-10T07:45:39.349 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:45:39.350 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T07:45:39.358 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T07:45:39.359 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T07:45:39.369 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T07:45:39.458 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T07:45:39.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:45:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:45:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.595 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T07:45:39.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T07:45:39.639 INFO:teuthology.task.internal:Duration was 1685.094029 seconds 2024-02-10T07:45:39.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T07:45:39.652 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T07:45:39.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T07:45:39.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T07:45:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:45:39.697 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:45:40.099 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T07:45:40.099 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T07:45:40.100 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T07:45:40.159 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-02-10T07:45:40.159 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T07:45:40.182 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T07:45:40.183 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T07:45:40.202 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T07:45:40.252 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T07:45:40.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T07:45:40.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T07:45:40.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T07:45:40.715 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T07:45:40.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T07:45:40.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T07:45:40.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T07:45:40.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:45:40.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:45:40.806 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-02-10T07:45:40.835 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-02-10T07:45:40.859 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:45:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.884 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:45:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T07:45:40.923 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T07:45:40.923 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/remote/smithi064 2024-02-10T07:45:40.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T07:45:40.970 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204/remote/smithi136 2024-02-10T07:45:40.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T07:45:41.010 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T07:45:41.011 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T07:45:41.013 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T07:45:41.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T07:45:41.070 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T07:45:41.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T07:45:41.080 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T07:45:41.080 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T07:45:41.082 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T07:45:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: 265904 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:45 /home/ubuntu/cephtest 2024-02-10T07:45:41.113 INFO:teuthology.orchestra.run.smithi136.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:45 /home/ubuntu/cephtest 2024-02-10T07:45:41.115 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T07:45:41.124 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T07:45:41.144 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T07:45:41.172 DEBUG:teuthology.nuke:shortname: smithi064 2024-02-10T07:45:41.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:45:41.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:40.582474', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6r+X1Nc51VPuwiRcnoeeWcCXT190kaw/sYXARFA9Cow+o1Fp1br3UFSmKFTNSfSrwqVbw8JsKpkseSY9Jz95I='} 2024-02-10T07:45:41.198 DEBUG:teuthology.nuke:shortname: smithi136 2024-02-10T07:45:41.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:45:41.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:40.583937', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxmpSynvc11KbCIh4D0t5/jWjiN0YDJnfffjInZO6k8Mqeq1T2+IifrX87mOIZ7LYzOVzRtr122ijg68/P/awQ='} 2024-02-10T07:45:41.246 INFO:teuthology.orchestra.console.smithi064:Power off 2024-02-10T07:45:41.246 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T07:45:41.271 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-02-10T07:45:41.271 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:45:41.287 INFO:teuthology.orchestra.console.smithi136:Power off 2024-02-10T07:45:41.287 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T07:45:41.297 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-10T07:45:41.311 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-02-10T07:45:41.311 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:45:41.333 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-02-10T07:45:45.298 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:45:45.333 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:45:45.407 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-10T07:45:45.441 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-02-10T07:45:49.412 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:45:49.442 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:45:49.522 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-02-10T07:45:49.554 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-02-10T07:45:49.624 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-02-10T07:45:49.654 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-02-10T07:45:49.759 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1685.094028711319 failure_reason: '"2024-02-10T07:26:33.709877+0000 mon.a (mon.0) 466 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T07:45:49.759 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T07:45:49.860 INFO:teuthology.run:FAIL