2024-04-07T20:37:12.445 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-07T20:37:12.446 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} --archive /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282 --verbose -- /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/orig.config.yaml 2024-04-07T20:37:12.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-07T20:37:12.560 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7634282' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d workunit: branch: main sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 7151 sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d sleep_before_teardown: 0 subset: 39/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d targets: smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOjRNcyFziMat0n9inS6w2GDfnJ+8G/WWtvjs0oGMcrS9h0s70dVGCJNHk03Ic8zck+Fw84cKvzp/wH6a7+vsho= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXVEGOu3DIVpWJjYDDZ5/MvQK9d7cePXfofX/S+0aDruk/NFwKjz8c1ZqolJ+b1FNWD/zbXMJZHGi04ddsgLeo= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-01_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-07T20:37:12.560 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa; will attempt to use it 2024-04-07T20:37:12.561 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks 2024-04-07T20:37:12.561 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-07T20:37:12.563 INFO:teuthology.task.internal:Checking packages... 2024-04-07T20:37:12.587 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e5c885fe95ee982ff7b45214daa61b1a6d09d57d' 2024-04-07T20:37:12.588 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-07T20:37:12.588 INFO:teuthology.packaging:ref: None 2024-04-07T20:37:12.588 INFO:teuthology.packaging:tag: None 2024-04-07T20:37:12.588 INFO:teuthology.packaging:branch: main 2024-04-07T20:37:12.588 INFO:teuthology.packaging:sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:37:12.588 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-07T20:37:12.959 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2778.ga971fd5f 2024-04-07T20:37:12.961 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-07T20:37:12.968 INFO:teuthology.task.internal:no buildpackages task found 2024-04-07T20:37:12.968 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-07T20:37:12.987 INFO:teuthology.task.internal:Saving configuration 2024-04-07T20:37:12.999 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-07T20:37:13.005 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-07T20:37:13.031 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 20:31:25.337603', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOjRNcyFziMat0n9inS6w2GDfnJ+8G/WWtvjs0oGMcrS9h0s70dVGCJNHk03Ic8zck+Fw84cKvzp/wH6a7+vsho='} 2024-04-07T20:37:13.054 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 20:31:25.339298', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXVEGOu3DIVpWJjYDDZ5/MvQK9d7cePXfofX/S+0aDruk/NFwKjz8c1ZqolJ+b1FNWD/zbXMJZHGi04ddsgLeo='} 2024-04-07T20:37:13.054 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-07T20:37:13.060 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-07T20:37:13.060 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-07T20:37:13.060 INFO:teuthology.run_tasks:Running task console_log... 2024-04-07T20:37:13.127 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0d084ef0d0>, signals=[15]) 2024-04-07T20:37:13.128 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-07T20:37:13.134 INFO:teuthology.task.internal:Opening connections... 2024-04-07T20:37:13.135 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-07T20:37:13.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:37:13.248 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-04-07T20:37:13.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:37:13.349 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-07T20:37:13.356 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-07T20:37:13.374 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-07T20:37:13.374 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-04-07T20:37:13.434 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-04-07T20:37:13.435 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-07T20:37:13.435 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-04-07T20:37:13.435 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-07T20:37:13.435 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-07T20:37:13.435 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-07T20:37:13.436 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-07T20:37:13.461 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-04-07T20:37:13.479 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-04-07T20:37:13.479 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-04-07T20:37:13.536 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-07T20:37:13.537 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-07T20:37:13.538 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-04-07T20:37:13.560 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-07T20:37:13.569 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-07T20:37:13.577 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-07T20:37:13.577 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-07T20:37:13.580 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-04-07T20:37:13.595 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-07T20:37:13.602 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-07T20:37:13.602 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-07T20:37:13.641 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-04-07T20:37:13.739 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-07T20:37:13.791 INFO:teuthology.run_tasks:Running task kernel... 2024-04-07T20:37:13.805 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-07T20:37:13.805 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-07T20:37:13.806 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-07T20:37:13.806 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-07T20:37:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T20:37:13.823 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-04-07T20:37:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T20:37:13.824 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-07T20:37:13.880 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-361.el9.x86_64 2024-04-07T20:37:13.881 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-361.el9.x86_64 2024-04-07T20:37:13.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-04-07T20:37:13.883 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-04-07T20:37:13.883 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2024-04-07T20:37:13.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-04-07T20:37:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 19 MB/s | 8.1 MB 00:00 2024-04-07T20:37:16.047 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 17 MB/s | 8.1 MB 00:00 2024-04-07T20:37:17.683 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.5 MB 00:00 2024-04-07T20:37:17.689 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.5 MB 00:00 2024-04-07T20:37:18.679 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-07T20:37:21.640 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-04-07T20:37:22.688 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-04-07T20:37:23.691 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-07T20:37:27.531 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.1 MB 00:01 2024-04-07T20:37:29.023 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-07T20:37:34.668 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 21 MB 00:05 2024-04-07T20:37:35.890 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 728 kB/s | 6.1 MB 00:08 2024-04-07T20:37:37.299 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-04-07T20:37:39.046 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-07T20:37:39.756 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-07T20:37:41.879 INFO:teuthology.orchestra.run.smithi187.stderr:Modular dependency problems: 2024-04-07T20:37:41.879 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 1: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 2: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 3: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 4: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 5: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 6: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 7: conflicting requests 2024-04-07T20:37:41.880 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 8: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 9: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 10: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 11: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 12: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 13: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 14: conflicting requests 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.881 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 15: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 16: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 17: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 18: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 19: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 20: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 21: conflicting requests 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.882 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 22: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 23: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 24: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 25: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 26: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 27: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 28: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 29: conflicting requests 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.883 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 30: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 31: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 32: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 33: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 34: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 35: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 36: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 37: conflicting requests 2024-04-07T20:37:41.884 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 38: cannot install the best candidate for the job 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 39: conflicting requests 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 40: cannot install the best candidate for the job 2024-04-07T20:37:41.885 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 41: cannot install the best candidate for the job 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 42: cannot install the best candidate for the job 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.886 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 43: cannot install the best candidate for the job 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 44: cannot install the best candidate for the job 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: Problem 45: cannot install the best candidate for the job 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-07T20:37:41.887 INFO:teuthology.orchestra.run.smithi187.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-07T20:37:42.264 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-07T20:37:42.264 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-07T20:37:42.378 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-07T20:37:42.383 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-07T20:37:42.383 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 76 M 2024-04-07T20:37:42.384 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 123 M 2024-04-07T20:37:42.385 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-07T20:37:42.716 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 331 kB/s | 66 kB 00:00 2024-04-07T20:37:44.150 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 9.1 MB/s | 15 MB 00:01 2024-04-07T20:37:44.300 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 6.3 kB/s | 1.7 kB 00:00 2024-04-07T20:37:44.750 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 14 MB/s | 27 MB 00:02 2024-04-07T20:37:45.292 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-04-07T20:37:45.292 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-07T20:37:45.292 INFO:teuthology.orchestra.run.smithi187.stdout:Total 26 MB/s | 76 MB 00:02 2024-04-07T20:37:45.954 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-07T20:37:46.301 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-07T20:37:46.301 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-07T20:37:46.437 INFO:teuthology.orchestra.run.smithi177.stderr:Modular dependency problems: 2024-04-07T20:37:46.437 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 1: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 2: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 3: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 4: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 5: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 6: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 7: conflicting requests 2024-04-07T20:37:46.438 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 8: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 9: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 10: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 11: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 12: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 13: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 14: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 15: conflicting requests 2024-04-07T20:37:46.439 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 16: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 17: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 18: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 19: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 20: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 21: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 22: conflicting requests 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.440 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 23: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 24: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 25: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 26: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 27: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 28: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 29: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 30: conflicting requests 2024-04-07T20:37:46.441 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 31: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 32: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 33: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 34: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 35: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 36: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 37: conflicting requests 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.442 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 38: cannot install the best candidate for the job 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 39: conflicting requests 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 40: cannot install the best candidate for the job 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.443 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 41: cannot install the best candidate for the job 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 42: cannot install the best candidate for the job 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 43: cannot install the best candidate for the job 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.444 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 44: cannot install the best candidate for the job 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: Problem 45: cannot install the best candidate for the job 2024-04-07T20:37:46.445 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.446 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.446 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.446 INFO:teuthology.orchestra.run.smithi177.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-07T20:37:46.891 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-07T20:37:46.891 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-07T20:37:46.996 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-07T20:37:47.008 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-07T20:37:47.008 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-04-07T20:37:47.008 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-07T20:37:47.008 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-04-07T20:37:47.008 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-04-07T20:37:47.009 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:37:47.010 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 76 M 2024-04-07T20:37:47.010 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 123 M 2024-04-07T20:37:47.010 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-07T20:37:47.331 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-07T20:37:47.331 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-07T20:37:47.493 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 189 kB/s | 66 kB 00:00 2024-04-07T20:37:49.786 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-07T20:37:50.517 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-07T20:37:50.600 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T20:37:53.360 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T20:37:53.493 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T20:37:57.442 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.4 MB/s | 15 MB 00:10 2024-04-07T20:37:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T20:37:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:01.725 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 2.3 MB/s | 33 MB 00:14 2024-04-07T20:38:03.028 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:12.299 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.1 MB/s | 27 MB 00:24 2024-04-07T20:38:12.299 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-07T20:38:12.299 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.0 MB/s | 76 MB 00:25 2024-04-07T20:38:12.960 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-07T20:38:13.320 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-07T20:38:13.320 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-07T20:38:14.336 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-07T20:38:14.336 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-07T20:38:16.805 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-07T20:38:17.601 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-07T20:38:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T20:38:20.397 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T20:38:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T20:38:25.220 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T20:38:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:30.072 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:32.124 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:32.124 INFO:teuthology.orchestra.run.smithi187.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-07T20:38:32.124 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:38:32.169 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:32.600 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:32.600 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-07T20:38:32.600 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T20:38:32.600 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:38:33.025 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-07T20:38:33.375 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T20:38:33.482 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T20:38:33.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T20:38:33.483 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-07T20:38:33.483 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-07T20:38:33.484 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-07T20:38:33.484 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-07T20:38:33.484 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2024-04-07T20:38:33.484 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T20:38:33.563 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T20:38:33.563 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T20:38:33.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2024-04-07T20:38:33.597 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.06 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 68.el9 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Group : Unspecified 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 8264254 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-07T20:38:33.598 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : builder@centos.org 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:38:33.599 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2024-04-07T20:38:33.600 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-07T20:38:33.600 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 5.14.0-432.el9.x86_64 2024-04-07T20:38:33.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-07T20:38:34.241 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-04-07T20:38:35.647 INFO:teuthology.orchestra.run.smithi187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-07T20:38:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-04-07T20:38:35.703 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-07T20:38:35.719 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.bfgvEXg5bf 2024-04-07T20:38:35.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bfgvEXg5bf 2024-04-07T20:38:35.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.bfgvEXg5bf 2024-04-07T20:38:35.927 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.bfgvEXg5bf is 6KB 2024-04-07T20:38:35.976 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.bfgvEXg5bf 2024-04-07T20:38:35.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2024-04-07T20:38:36.056 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-07T20:38:36.056 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-07T20:38:36.056 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-07T20:38:36.056 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-07T20:38:36.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-07T20:38:36.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-04-07T20:38:59.654 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:38:59.655 INFO:teuthology.orchestra.run.smithi177.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-07T20:38:59.655 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:38:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:39:00.061 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:39:00.061 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-07T20:39:00.061 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T20:39:00.061 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T20:39:00.486 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T20:39:00.486 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:39:00.486 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-04-07T20:39:00.486 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-07T20:39:00.487 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-07T20:39:00.487 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-07T20:39:00.487 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-07T20:39:00.487 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:39:00.487 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-07T20:39:00.830 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T20:39:00.971 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T20:39:00.971 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T20:39:00.971 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-07T20:39:00.971 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-07T20:39:00.971 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-07T20:39:00.971 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-07T20:39:00.972 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-04-07T20:39:00.972 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T20:39:01.052 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T20:39:01.052 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T20:39:01.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-04-07T20:39:01.094 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-07T20:39:01.095 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:39:01.096 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-04-07T20:39:01.097 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-07T20:39:01.097 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-432.el9.x86_64 2024-04-07T20:39:01.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-07T20:39:01.718 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-04-07T20:39:03.172 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-07T20:39:03.206 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-04-07T20:39:03.208 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-07T20:39:03.229 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.2rM8tFdTNP 2024-04-07T20:39:03.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2rM8tFdTNP 2024-04-07T20:39:03.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.2rM8tFdTNP 2024-04-07T20:39:03.436 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.2rM8tFdTNP is 6KB 2024-04-07T20:39:03.485 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.2rM8tFdTNP 2024-04-07T20:39:03.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-04-07T20:39:03.564 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-07T20:39:03.564 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-07T20:39:03.564 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-07T20:39:03.564 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-07T20:39:03.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-07T20:39:03.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-04-07T20:39:06.228 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-07T20:39:06.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-07T20:39:06.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:39:24.703 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-07T20:39:33.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-07T20:39:33.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:39:33.721 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-07T20:39:33.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-07T20:39:33.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:39:36.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-07T20:39:48.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-07T20:39:48.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:39:52.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-04-07T20:40:01.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-07T20:40:01.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:40:04.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-04-07T20:40:16.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-07T20:40:16.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:40:20.613 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-07T20:40:21.060 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-07T20:40:21.060 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-04-07T20:40:21.060 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-07T20:40:21.075 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-432.el9.x86_64 2024-04-07T20:40:21.075 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-07T20:40:21.075 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-07T20:40:21.075 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-07T20:40:22.076 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-07T20:40:22.076 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-07T20:40:22.153 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-04-07T20:40:22.195 DEBUG:teuthology.parallel:result is None 2024-04-07T20:40:47.912 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-07T20:40:48.308 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-07T20:40:48.308 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-04-07T20:40:48.309 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-04-07T20:40:48.322 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-432.el9.x86_64 2024-04-07T20:40:48.322 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-07T20:40:48.322 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-07T20:40:48.322 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-07T20:40:49.323 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-07T20:40:49.323 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-07T20:40:49.388 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-04-07T20:40:49.427 DEBUG:teuthology.parallel:result is None 2024-04-07T20:40:49.428 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-07T20:40:49.435 INFO:teuthology.task.internal:Creating test directory... 2024-04-07T20:40:49.435 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-07T20:40:49.437 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-07T20:40:49.456 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-07T20:40:49.508 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-07T20:40:49.534 INFO:teuthology.task.internal:Creating archive directory... 2024-04-07T20:40:49.534 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-07T20:40:49.536 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-07T20:40:49.578 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-07T20:40:49.596 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-07T20:40:49.596 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-07T20:40:49.614 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-07T20:40:49.645 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T20:40:49.657 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T20:40:49.658 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T20:40:49.672 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T20:40:49.674 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-07T20:40:49.682 INFO:teuthology.task.internal:Configuring sudo... 2024-04-07T20:40:49.682 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-07T20:40:49.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-07T20:40:49.736 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-07T20:40:49.745 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-07T20:40:49.745 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-07T20:40:49.778 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-07T20:40:49.794 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T20:40:49.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T20:40:49.933 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T20:40:49.975 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T20:40:50.016 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:40:50.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-07T20:40:50.083 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T20:40:50.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T20:40:50.185 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T20:40:50.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T20:40:50.313 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:40:50.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-07T20:40:50.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-07T20:40:50.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-04-07T20:40:50.461 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T20:40:50.468 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T20:40:50.841 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-07T20:40:50.849 INFO:teuthology.task.internal:Starting timer... 2024-04-07T20:40:50.849 INFO:teuthology.run_tasks:Running task pcp... 2024-04-07T20:40:50.908 INFO:teuthology.run_tasks:Running task selinux... 2024-04-07T20:40:50.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-04-07T20:40:51.039 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-04-07T20:40:51.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-04-07T20:40:51.113 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-04-07T20:40:51.115 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-07T20:40:51.115 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-04-07T20:40:51.162 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-04-07T20:40:51.162 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-04-07T20:40:51.187 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-04-07T20:40:51.188 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi177.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-04-07T20:40:51.188 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-07T20:40:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T20:40:51.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-07T20:40:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T20:40:51.271 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-07T20:40:51.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-04-07T20:40:51.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-04-07T20:40:51.355 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-07T20:40:51.366 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-07T20:40:51.371 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-07T20:40:51.528 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-07T20:40:51.543 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-07T20:40:51.545 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi177.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-07T20:45:33.618 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi177.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-04-07T20:45:33.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-07T20:45:33.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:45:33.713 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-07T20:45:33.789 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-07T20:45:33.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-07T20:45:33.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T20:45:33.876 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-07T20:45:33.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-07T20:45:33.961 INFO:teuthology.run_tasks:Running task clock... 2024-04-07T20:45:33.971 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-07T20:45:33.971 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-07T20:45:33.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T20:45:33.974 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-07T20:45:33.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T20:45:34.006 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-07T20:45:34.024 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-07T20:45:34.032 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-07T20:45:34.051 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-07T20:45:34.055 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-04-07T20:45:34.069 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-04-07T20:45:34.080 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-04-07T20:45:34.085 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-07T20:45:34.094 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-04-07T20:45:34.102 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-07T20:45:34.109 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-07T20:45:34.126 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-07T20:45:34.162 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-07T20:45:34.165 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T20:45:34.165 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-07T20:45:34.165 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.166 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.166 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.166 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.182 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-07T20:45:34.185 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T20:45:34.185 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-07T20:45:34.185 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.185 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.185 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.185 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T20:45:34.186 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-07T20:45:34.284 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e5c885fe95ee982ff7b45214daa61b1a6d09d57d'} 2024-04-07T20:45:34.285 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-07T20:45:34.285 INFO:tasks.cephadm:Cluster fsid is c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:45:34.285 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-07T20:45:34.285 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.177', 'mon.c': '[v2:172.21.15.177:3301,v1:172.21.15.177:6790]', 'mon.b': '172.21.15.187'} 2024-04-07T20:45:34.286 INFO:tasks.cephadm:First mon is mon.a on smithi177 2024-04-07T20:45:34.286 INFO:tasks.cephadm:First mgr is y 2024-04-07T20:45:34.286 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-07T20:45:34.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-04-07T20:45:34.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2024-04-07T20:45:34.342 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-07T20:45:34.342 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-07T20:45:34.577 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 7 20:45 /home/ubuntu/cephtest/cephadm 2024-04-07T20:45:34.578 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-07T20:45:34.761 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 7 20:45 /home/ubuntu/cephtest/cephadm 2024-04-07T20:45:34.762 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-07T20:45:34.778 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-07T20:45:34.803 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-07T20:45:34.803 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-07T20:45:34.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-07T20:45:35.664 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-07T20:45:35.771 INFO:teuthology.orchestra.run.smithi187.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout: "repo_digests": [ 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-04-07T20:45:53.891 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-07T20:45:54.507 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:45:54.524 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-04-07T20:45:54.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2024-04-07T20:45:54.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-04-07T20:45:54.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2024-04-07T20:45:54.665 INFO:tasks.cephadm:Writing seed config... 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-07T20:45:54.667 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-07T20:45:54.668 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:45:54.668 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-07T20:45:54.689 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c7f93d4c-f51f-11ee-b649-cb9ed24678a4 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-07T20:45:54.689 DEBUG:teuthology.orchestra.run.smithi177:mon.a> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a.service 2024-04-07T20:45:54.731 DEBUG:teuthology.orchestra.run.smithi177:mgr.y> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y.service 2024-04-07T20:45:54.774 INFO:tasks.cephadm:Bootstrapping... 2024-04-07T20:45:54.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.177 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-07T20:45:54.968 INFO:teuthology.orchestra.run.smithi177.stderr:-------------------------------------------------------------------------------- 2024-04-07T20:45:54.968 INFO:teuthology.orchestra.run.smithi177.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'c7f93d4c-f51f-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.177', '--skip-admin-label'] 2024-04-07T20:45:54.990 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: 4.9.4-dev 2024-04-07T20:45:54.993 INFO:teuthology.orchestra.run.smithi177.stderr:Verifying podman|docker is present... 2024-04-07T20:45:55.015 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: 4.9.4-dev 2024-04-07T20:45:55.017 INFO:teuthology.orchestra.run.smithi177.stderr:Verifying lvm2 is present... 2024-04-07T20:45:55.017 INFO:teuthology.orchestra.run.smithi177.stderr:Verifying time synchronization is in place... 2024-04-07T20:45:55.023 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-07T20:45:55.029 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: inactive 2024-04-07T20:45:55.036 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: enabled 2024-04-07T20:45:55.042 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: active 2024-04-07T20:45:55.043 INFO:teuthology.orchestra.run.smithi177.stderr:Unit chronyd.service is enabled and running 2024-04-07T20:45:55.043 INFO:teuthology.orchestra.run.smithi177.stderr:Repeating the final host check... 2024-04-07T20:45:55.064 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: 4.9.4-dev 2024-04-07T20:45:55.066 INFO:teuthology.orchestra.run.smithi177.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-07T20:45:55.066 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl is present 2024-04-07T20:45:55.066 INFO:teuthology.orchestra.run.smithi177.stderr:lvcreate is present 2024-04-07T20:45:55.072 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-07T20:45:55.078 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: inactive 2024-04-07T20:45:55.084 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: enabled 2024-04-07T20:45:55.090 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: active 2024-04-07T20:45:55.091 INFO:teuthology.orchestra.run.smithi177.stderr:Unit chronyd.service is enabled and running 2024-04-07T20:45:55.091 INFO:teuthology.orchestra.run.smithi177.stderr:Host looks OK 2024-04-07T20:45:55.091 INFO:teuthology.orchestra.run.smithi177.stderr:Cluster fsid: c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:45:55.091 INFO:teuthology.orchestra.run.smithi177.stderr:Acquiring lock 139930327593360 on /run/cephadm/c7f93d4c-f51f-11ee-b649-cb9ed24678a4.lock 2024-04-07T20:45:55.091 INFO:teuthology.orchestra.run.smithi177.stderr:Lock 139930327593360 acquired on /run/cephadm/c7f93d4c-f51f-11ee-b649-cb9ed24678a4.lock 2024-04-07T20:45:55.091 INFO:teuthology.orchestra.run.smithi177.stderr:Verifying IP 172.21.15.177 port 3300 ... 2024-04-07T20:45:55.092 INFO:teuthology.orchestra.run.smithi177.stderr:Verifying IP 172.21.15.177 port 6789 ... 2024-04-07T20:45:55.092 INFO:teuthology.orchestra.run.smithi177.stderr:Base mon IP is 172.21.15.177, final addrv is [v2:172.21.15.177:3300,v1:172.21.15.177:6789] 2024-04-07T20:45:55.094 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.177 metric 100 2024-04-07T20:45:55.094 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.177 metric 100 2024-04-07T20:45:55.097 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-07T20:45:55.097 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-07T20:45:55.097 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-04-07T20:45:55.099 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: inet6 fe80::ec4:7aff:feea:b04b/64 scope link 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:Mon IP `172.21.15.177` is in CIDR network `172.21.0.0/20` 2024-04-07T20:45:55.100 INFO:teuthology.orchestra.run.smithi177.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-07T20:45:55.101 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-07T20:45:55.201 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-07T20:45:55.525 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Getting image source signatures 2024-04-07T20:45:55.525 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-07T20:45:55.525 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-07T20:45:55.525 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-07T20:45:55.526 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-07T20:45:55.526 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-07T20:45:55.526 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-07T20:45:55.686 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: Writing manifest to image destination 2024-04-07T20:45:55.847 INFO:teuthology.orchestra.run.smithi177.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-07T20:45:56.257 INFO:teuthology.orchestra.run.smithi177.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-07T20:45:56.779 INFO:teuthology.orchestra.run.smithi177.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-07T20:45:56.779 INFO:teuthology.orchestra.run.smithi177.stderr:Extracting ceph user uid/gid from container image... 2024-04-07T20:45:57.204 INFO:teuthology.orchestra.run.smithi177.stderr:stat: 167 167 2024-04-07T20:45:57.585 INFO:teuthology.orchestra.run.smithi177.stderr:Creating initial keys... 2024-04-07T20:45:57.968 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph-authtool: AQAFBhNmX2EiNRAAb38JolLvSibEn6bGZlKA4g== 2024-04-07T20:45:58.764 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph-authtool: AQAGBhNmbk6MKRAAkPZQNscgWDl8jBz5lMnzXA== 2024-04-07T20:45:59.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph-authtool: AQAHBhNmoHN9GxAA9rBUpQTgqDWWYeFWvz1uUQ== 2024-04-07T20:45:59.928 INFO:teuthology.orchestra.run.smithi177.stderr:Creating initial monmap... 2024-04-07T20:46:00.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-07T20:46:00.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-07T20:46:00.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:00.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-07T20:46:00.714 INFO:teuthology.orchestra.run.smithi177.stderr:monmaptool for a [v2:172.21.15.177:3300,v1:172.21.15.177:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-07T20:46:00.714 INFO:teuthology.orchestra.run.smithi177.stderr:setting min_mon_release = octopus 2024-04-07T20:46:00.714 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/monmaptool: set fsid to c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:00.714 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-07T20:46:00.714 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:46:00.714 INFO:teuthology.orchestra.run.smithi177.stderr:Creating mon... 2024-04-07T20:46:01.569 INFO:teuthology.orchestra.run.smithi177.stderr:create mon.a on 2024-04-07T20:46:01.788 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-07T20:46:01.935 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4.target. 2024-04-07T20:46:01.936 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4.target. 2024-04-07T20:46:02.246 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Failed to reset failed state of unit ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a.service: Unit ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a.service not loaded. 2024-04-07T20:46:02.254 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4.target.wants/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@.service. 2024-04-07T20:46:02.921 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:02 smithi177 ceph-mon[24512]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-07T20:46:02.934 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: disabled 2024-04-07T20:46:02.941 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: inactive 2024-04-07T20:46:02.941 INFO:teuthology.orchestra.run.smithi177.stderr:firewalld.service is not enabled 2024-04-07T20:46:02.941 INFO:teuthology.orchestra.run.smithi177.stderr:Not possible to enable service . firewalld.service is not available 2024-04-07T20:46:02.942 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for mon to start... 2024-04-07T20:46:02.942 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for mon... 2024-04-07T20:46:03.176 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:02 smithi177 bash[24493]: ec58076dd749b210da702ddcde100302764bf407ccc10a15b778e6f9929f7893 2024-04-07T20:46:03.177 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:02 smithi177 systemd[1]: Started Ceph mon.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:46:03.423 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: cluster: 2024-04-07T20:46:03.423 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: id: c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:03.423 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-07T20:46:03.423 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: services: 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.529195s) 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: data: 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: pgs: 2024-04-07T20:46:03.424 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:03.938 INFO:teuthology.orchestra.run.smithi177.stderr:mon is available 2024-04-07T20:46:03.938 INFO:teuthology.orchestra.run.smithi177.stderr:Assimilating anything we can from ceph.conf... 2024-04-07T20:46:04.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:03 smithi177 ceph-mon[24512]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-07T20:46:04.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:03 smithi177 ceph-mon[24512]: monmap e1: 1 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} 2024-04-07T20:46:04.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:03 smithi177 ceph-mon[24512]: fsmap 2024-04-07T20:46:04.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:03 smithi177 ceph-mon[24512]: osdmap e1: 0 total, 0 up, 0 in 2024-04-07T20:46:04.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:03 smithi177 ceph-mon[24512]: mgrmap e1: no daemons active 2024-04-07T20:46:04.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:03 smithi177 ceph-mon[24512]: from='client.? 172.21.15.177:0/2387217948' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: [global] 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: fsid = c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.177:3300,v1:172.21.15.177:6789] 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:04.390 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: [mgr] 2024-04-07T20:46:04.391 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = True 2024-04-07T20:46:04.391 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-07T20:46:04.391 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:04.391 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: [osd] 2024-04-07T20:46:04.391 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-07T20:46:04.391 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-07T20:46:04.910 INFO:teuthology.orchestra.run.smithi177.stderr:Generating new minimal ceph.conf... 2024-04-07T20:46:05.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:04 smithi177 ceph-mon[24512]: from='client.? 172.21.15.177:0/799900815' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-07T20:46:05.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:04 smithi177 ceph-mon[24512]: from='client.? 172.21.15.177:0/799900815' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-07T20:46:05.899 INFO:teuthology.orchestra.run.smithi177.stderr:Restarting the monitor... 2024-04-07T20:46:05.918 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:05 smithi177 ceph-mon[24512]: from='client.? 172.21.15.177:0/3848753873' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:46:06.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:05 smithi177 systemd[1]: Stopping Ceph mon.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:46:06.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:05 smithi177 bash[24730]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.a" found: no such container 2024-04-07T20:46:06.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a[24508]: 2024-04-07T20:46:06.050+0000 7f6af18eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:46:06.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a[24508]: 2024-04-07T20:46:06.051+0000 7f6af18eb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-07T20:46:06.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 podman[24740]: 2024-04-07 20:46:06.113011734 +0000 UTC m=+0.159166805 container died ec58076dd749b210da702ddcde100302764bf407ccc10a15b778e6f9929f7893 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, maintainer=Guillaume Abrioux , architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T20:46:06.583 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 podman[24740]: 2024-04-07 20:46:06.45067116 +0000 UTC m=+0.496826239 container cleanup ec58076dd749b210da702ddcde100302764bf407ccc10a15b778e6f9929f7893 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, io.openshift.expose-services=, vcs-type=git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T20:46:06.583 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 bash[24740]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 podman[24754]: 2024-04-07 20:46:06.585028039 +0000 UTC m=+0.466972484 container remove ec58076dd749b210da702ddcde100302764bf407ccc10a15b778e6f9929f7893 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, vcs-type=git, GIT_CLEAN=True, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, release=754, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8) 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 bash[24766]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.a" found: no such container 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 systemd[1]: Stopped Ceph mon.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 systemd[1]: Starting Ceph mon.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 podman[24833]: 2024-04-07 20:46:06.819094113 +0000 UTC m=+0.089739041 container create f4208a5ace0f8ced6ed50662f402915d4d4ed2911ecb90c7f23e5a927af41a91 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, release=754, io.openshift.expose-services=, ceph=True, name=centos-stream) 2024-04-07T20:46:06.843 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:06 smithi177 podman[24833]: 2024-04-07 20:46:06.747790893 +0000 UTC m=+0.018435791 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-07T20:46:07.073 INFO:teuthology.orchestra.run.smithi177.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 podman[24833]: 2024-04-07 20:46:07.004827991 +0000 UTC m=+0.275472866 container init f4208a5ace0f8ced6ed50662f402915d4d4ed2911ecb90c7f23e5a927af41a91 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, release=754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc.) 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 podman[24833]: 2024-04-07 20:46:07.008022438 +0000 UTC m=+0.278667310 container start f4208a5ace0f8ced6ed50662f402915d4d4ed2911ecb90c7f23e5a927af41a91 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., ceph=True, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, name=centos-stream, distribution-scope=public, version=8) 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: set uid:gid to 167:167 (ceph:ceph) 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: pidfile_write: ignore empty --pid-file 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: load: jerasure load: lrc 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: RocksDB version: 6.15.5 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Compile date Apr 18 2022 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: DB SUMMARY 2024-04-07T20:46:07.104 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: DB Session ID: UYLIJX8JHRN3537UVFS5 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: CURRENT file: CURRENT 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: IDENTITY file: IDENTITY 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82657 ; 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.error_if_exists: 0 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.create_if_missing: 0 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.paranoid_checks: 1 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.env: 0x564e9b4c6860 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.fs: Posix File System 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.info_log: 0x564e9da83dc0 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_file_opening_threads: 16 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.statistics: (nil) 2024-04-07T20:46:07.105 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.use_fsync: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_log_file_size: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.keep_log_file_num: 1000 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.recycle_log_file_num: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.allow_fallocate: 1 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.allow_mmap_reads: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.allow_mmap_writes: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.use_direct_reads: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.create_missing_column_families: 0 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.db_log_dir: 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-07T20:46:07.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-07T20:46:07.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-07T20:46:07.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-07T20:46:07.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-07T20:46:07.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-07T20:46:07.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-07T20:46:07.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.advise_random_on_open: 1 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.db_write_buffer_size: 0 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.write_buffer_manager: 0x564e9db74240 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-07T20:46:07.108 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.rate_limiter: (nil) 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.wal_recovery_mode: 2 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.enable_thread_tracking: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.enable_pipelined_write: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.unordered_write: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.row_cache: None 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.wal_filter: None 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.allow_ingest_behind: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.preserve_deletes: 0 2024-04-07T20:46:07.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.two_write_queues: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.manual_wal_flush: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.atomic_flush: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.log_readahead_size: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.best_efforts_recovery: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.allow_data_in_errors: 0 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.db_host_id: __hostname__ 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_background_jobs: 2 2024-04-07T20:46:07.110 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_background_compactions: -1 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_subcompactions: 1 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_total_wal_size: 0 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_open_files: -1 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bytes_per_sync: 0 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-07T20:46:07.111 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_readahead_size: 0 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_background_flushes: -1 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Compression algorithms supported: 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kZSTD supported: 0 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kXpressCompression supported: 0 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kLZ4HCCompression supported: 1 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kLZ4Compression supported: 1 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kBZip2Compression supported: 0 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kZlibCompression supported: 1 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: kSnappyCompression supported: 1 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-07T20:46:07.112 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.merge_operator: 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_filter: None 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_filter_factory: None 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.sst_partitioner_factory: None 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564e9da4fd00) 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: index_type: 0 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: data_block_index_type: 0 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: index_shortening: 1 2024-04-07T20:46:07.113 INFO:journalctl@ceph.mon.a.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: hash_index_allow_collision: 1 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: checksum: 1 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: no_block_cache: 0 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_cache: 0x564e9daba170 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_cache_options: 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: capacity : 536870912 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: num_shard_bits : 4 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: strict_capacity_limit : 0 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_cache_compressed: (nil) 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: persistent_cache: (nil) 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_size: 4096 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_size_deviation: 10 2024-04-07T20:46:07.114 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_restart_interval: 16 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: index_block_restart_interval: 1 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: metadata_block_size: 4096 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: partition_filters: 0 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: use_delta_encoding: 1 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: whole_key_filtering: 1 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: verify_compression: 0 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: format_version: 4 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: enable_index_compression: 1 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout: block_align: 0 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.write_buffer_size: 33554432 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_write_buffer_number: 2 2024-04-07T20:46:07.115 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression: NoCompression 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression: Disabled 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.prefix_extractor: nullptr 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.num_levels: 7 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-07T20:46:07.116 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.level: 32767 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.strategy: 0 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compression_opts.enabled: false 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.target_file_size_base: 67108864 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-07T20:46:07.117 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.arena_block_size: 4194304 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.disable_auto_compactions: 0 2024-04-07T20:46:07.118 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.table_properties_collectors: 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.inplace_update_support: 0 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-07T20:46:07.119 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.bloom_locality: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.max_successive_merges: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.paranoid_file_checks: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.force_consistency_checks: 1 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.report_bg_io_stats: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.ttl: 2592000 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.enable_blob_files: false 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.min_blob_size: 0 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.blob_file_size: 268435456 2024-04-07T20:46:07.120 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712522767033333, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712522767034074, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79207, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77470, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71779, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712522767, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c373d226-daff-44ad-824f-b60d9b364a9c", "db_session_id": "UYLIJX8JHRN3537UVFS5"}} 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712522767034397, "job": 1, "event": "recovery_finished"} 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x564e9daa0a80 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: DB pointer 0x564e9dab0000 2024-04-07T20:46:07.121 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** DB Stats ** 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.122 INFO:journalctl@ceph.mon.a.smithi177.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Cumulative compaction: 0.00 GB write, 25.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Interval compaction: 0.00 GB write, 25.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T20:46:07.123 INFO:journalctl@ceph.mon.a.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.7 0.00 0.00 1 0.001 0 0 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T20:46:07.124 INFO:journalctl@ceph.mon.a.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: Cumulative compaction: 0.00 GB write, 25.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: starting mon.a rank 0 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???) e1 preinit fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).mds e1 new map 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).mds e1 print_map 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: e1 2024-04-07T20:46:07.125 INFO:journalctl@ceph.mon.a.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout: legacy client fscid: -1 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout: 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout: No filesystems configured 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expand map: {default=false} 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta from 'false' to 'false' 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expanded map: {default=false} 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expand map: {default=info} 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta from 'info' to 'info' 2024-04-07T20:46:07.126 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expanded map: {default=info} 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expand map: {default=daemon} 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expanded map: {default=daemon} 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expand map: {default=debug} 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta from 'debug' to 'debug' 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: expand_channel_meta expanded map: {default=debug} 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 bash[24833]: f4208a5ace0f8ced6ed50662f402915d4d4ed2911ecb90c7f23e5a927af41a91 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 systemd[1]: Started Ceph mon.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: monmap e1: 1 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: fsmap 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: osdmap e1: 0 total, 0 up, 0 in 2024-04-07T20:46:07.127 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:07 smithi177 ceph-mon[24852]: mgrmap e1: no daemons active 2024-04-07T20:46:08.037 INFO:teuthology.orchestra.run.smithi177.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-07T20:46:08.037 INFO:teuthology.orchestra.run.smithi177.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-07T20:46:08.037 INFO:teuthology.orchestra.run.smithi177.stderr:Creating mgr... 2024-04-07T20:46:08.037 INFO:teuthology.orchestra.run.smithi177.stderr:Verifying port 9283 ... 2024-04-07T20:46:08.202 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Failed to reset failed state of unit ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y.service: Unit ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y.service not loaded. 2024-04-07T20:46:08.209 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4.target.wants/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y.service → /etc/systemd/system/ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@.service. 2024-04-07T20:46:08.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 systemd[1]: Starting Ceph mgr.y for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:46:08.854 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:08 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2080327665' entity='client.admin' 2024-04-07T20:46:08.855 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 podman[25070]: 2024-04-07 20:46:08.486368443 +0000 UTC m=+0.018018099 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-07T20:46:08.855 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 podman[25070]: 2024-04-07 20:46:08.59519338 +0000 UTC m=+0.126843036 container create f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y, vendor=Red Hat, Inc., ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD) 2024-04-07T20:46:08.855 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 podman[25070]: 2024-04-07 20:46:08.764325966 +0000 UTC m=+0.295975622 container init f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, maintainer=Guillaume Abrioux , vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, RELEASE=HEAD) 2024-04-07T20:46:08.855 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 podman[25070]: 2024-04-07 20:46:08.766817614 +0000 UTC m=+0.298467277 container start f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.openshift.expose-services=, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, vcs-type=git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, release=754) 2024-04-07T20:46:08.855 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 bash[25070]: f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 2024-04-07T20:46:08.869 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: disabled 2024-04-07T20:46:08.878 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: inactive 2024-04-07T20:46:08.878 INFO:teuthology.orchestra.run.smithi177.stderr:firewalld.service is not enabled 2024-04-07T20:46:08.878 INFO:teuthology.orchestra.run.smithi177.stderr:Not possible to enable service . firewalld.service is not available 2024-04-07T20:46:08.888 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: disabled 2024-04-07T20:46:08.896 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: inactive 2024-04-07T20:46:08.897 INFO:teuthology.orchestra.run.smithi177.stderr:firewalld.service is not enabled 2024-04-07T20:46:08.897 INFO:teuthology.orchestra.run.smithi177.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-07T20:46:08.897 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for mgr to start... 2024-04-07T20:46:08.897 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for mgr... 2024-04-07T20:46:09.132 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:08 smithi177 systemd[1]: Started Ceph mgr.y for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:46:09.402 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:09.265+0000 7f6ee6dae000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:46:09.402 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:09.358+0000 7f6ee6dae000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:46:09.406 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:09.406 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:09.406 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsid": "c7f93d4c-f51f-11ee-b649-cb9ed24678a4", 2024-04-07T20:46:09.406 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "health": { 2024-04-07T20:46:09.406 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T20:46:09.406 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 0 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "a" 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T20:46:09.407 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T20:46:09.408 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T20:46:09.412 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T20:46:09.413 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "available": false, 2024-04-07T20:46:09.414 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "iostat", 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "nfs", 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "restful" 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modified": "2024-04-07T20:46:02.895031+0000", 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:09.415 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:09.416 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T20:46:09.416 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:09.735 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:09 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4130254388' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T20:46:10.002 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:09.967+0000 7f6ee6dae000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:46:10.004 INFO:teuthology.orchestra.run.smithi177.stderr:mgr not available, waiting (1/15)... 2024-04-07T20:46:10.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.056+0000 7f6ee6dae000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:46:10.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.137+0000 7f6ee6dae000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:46:10.552 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.285+0000 7f6ee6dae000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:46:10.553 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.365+0000 7f6ee6dae000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:46:10.553 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.471+0000 7f6ee6dae000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:46:10.553 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.553+0000 7f6ee6dae000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:46:11.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.715+0000 7f6ee6dae000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:46:11.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:10.799+0000 7f6ee6dae000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:46:12.130 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:11 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:11.812+0000 7f6ee6dae000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:46:12.130 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:11 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:11.941+0000 7f6ee6dae000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:46:12.396 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:12.138+0000 7f6ee6dae000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:46:12.396 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:12.224+0000 7f6ee6dae000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:46:12.396 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:12.364+0000 7f6ee6dae000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:46:12.514 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:12.514 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:12.514 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsid": "c7f93d4c-f51f-11ee-b649-cb9ed24678a4", 2024-04-07T20:46:12.514 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "health": { 2024-04-07T20:46:12.514 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 0 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "a" 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T20:46:12.515 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.516 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T20:46:12.516 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:12.516 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T20:46:12.516 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T20:46:12.516 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T20:46:12.516 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T20:46:12.519 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "available": false, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "iostat", 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "nfs", 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "restful" 2024-04-07T20:46:12.520 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modified": "2024-04-07T20:46:02.895031+0000", 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T20:46:12.521 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:12.994 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:12.688+0000 7f6ee6dae000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:46:12.994 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:12.911+0000 7f6ee6dae000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:46:12.994 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:12.994+0000 7f6ee6dae000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:46:13.094 INFO:teuthology.orchestra.run.smithi177.stderr:mgr not available, waiting (2/15)... 2024-04-07T20:46:13.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:13 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2147847463' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T20:46:13.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:13.167+0000 7f6ee6dae000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:46:14.366 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:14.078+0000 7f6ee6dae000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:46:14.670 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:14.366+0000 7f6ee6dae000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:46:14.670 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:14.459+0000 7f6ee6dae000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:46:14.670 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:14.606+0000 7f6ee6dae000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:46:15.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: Activating manager daemon y 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: mgrmap e2: y(active, starting, since 0.002722s) 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: Manager daemon y is now available 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' 2024-04-07T20:46:15.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:14 smithi177 ceph-mon[24852]: from='mgr.14100 172.21.15.177:0/3200362089' entity='mgr.y' 2024-04-07T20:46:15.559 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:15.559 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:15.559 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsid": "c7f93d4c-f51f-11ee-b649-cb9ed24678a4", 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "health": { 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 0 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "a" 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T20:46:15.560 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T20:46:15.561 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T20:46:15.564 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T20:46:15.564 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T20:46:15.564 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.564 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T20:46:15.565 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "available": false, 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "iostat", 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "nfs", 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "restful" 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modified": "2024-04-07T20:46:02.895031+0000", 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T20:46:15.566 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:16.010 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:15 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3797862540' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T20:46:16.010 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:15 smithi177 ceph-mon[24852]: mgrmap e3: y(active, since 1.00573s) 2024-04-07T20:46:16.074 INFO:teuthology.orchestra.run.smithi177.stderr:mgr not available, waiting (3/15)... 2024-04-07T20:46:18.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:17 smithi177 ceph-mon[24852]: mgrmap e4: y(active, since 2s) 2024-04-07T20:46:22.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:21 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3352999996' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsid": "c7f93d4c-f51f-11ee-b649-cb9ed24678a4", 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "health": { 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T20:46:22.305 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 0 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "a" 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T20:46:22.306 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T20:46:22.307 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "available": true, 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "iostat", 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "nfs", 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "restful" 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ], 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T20:46:22.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T20:46:22.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "modified": "2024-04-07T20:46:02.895031+0000", 2024-04-07T20:46:22.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "services": {} 2024-04-07T20:46:22.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: }, 2024-04-07T20:46:22.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T20:46:22.309 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:27.415 INFO:teuthology.orchestra.run.smithi177.stderr:mgr is available 2024-04-07T20:46:27.997 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:27.997 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: [global] 2024-04-07T20:46:27.997 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: fsid = c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:27.997 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: [mgr] 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: [osd] 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-07T20:46:27.998 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-07T20:46:28.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:28 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1574318281' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-07T20:46:28.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:28 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1574318281' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-07T20:46:28.571 INFO:teuthology.orchestra.run.smithi177.stderr:Enabling cephadm module... 2024-04-07T20:46:29.725 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:29 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4164368006' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-07T20:46:29.725 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:29 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setuser ceph since I am not root 2024-04-07T20:46:29.725 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:29 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setgroup ceph since I am not root 2024-04-07T20:46:30.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:29 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:29.769+0000 7fcb5b43c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:46:30.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:29 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:29.862+0000 7fcb5b43c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:46:30.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4164368006' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-07T20:46:30.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-mon[24852]: mgrmap e5: y(active, since 14s) 2024-04-07T20:46:30.474 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:30.474 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-07T20:46:30.474 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "available": true, 2024-04-07T20:46:30.474 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-07T20:46:30.474 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-07T20:46:30.474 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:30.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:30.468+0000 7fcb5b43c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:46:30.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:30.565+0000 7fcb5b43c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:46:30.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:30.646+0000 7fcb5b43c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:46:31.067 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:30.797+0000 7fcb5b43c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:46:31.067 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:30.876+0000 7fcb5b43c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:46:31.067 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:30.985+0000 7fcb5b43c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:46:31.067 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:31 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:31.068+0000 7fcb5b43c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:46:31.192 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for the mgr to restart... 2024-04-07T20:46:31.192 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for mgr epoch 5... 2024-04-07T20:46:31.322 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:31 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3075422147' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-07T20:46:31.322 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:31 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:31.235+0000 7fcb5b43c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:46:31.322 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:31 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:31.323+0000 7fcb5b43c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:46:32.665 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:32.351+0000 7fcb5b43c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:46:32.665 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:32.478+0000 7fcb5b43c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:46:32.665 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:32.666+0000 7fcb5b43c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:46:33.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:32.753+0000 7fcb5b43c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:46:33.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:32.888+0000 7fcb5b43c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:46:33.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:33.208+0000 7fcb5b43c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:46:33.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:33.430+0000 7fcb5b43c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:46:34.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:33.514+0000 7fcb5b43c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:46:34.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:33.683+0000 7fcb5b43c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:46:34.879 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:34 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:34.590+0000 7fcb5b43c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:46:34.879 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:34 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:34.879+0000 7fcb5b43c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:46:35.188 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:34 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:34.972+0000 7fcb5b43c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:46:35.188 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:35 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:35.121+0000 7fcb5b43c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:46:35.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:35 smithi177 ceph-mon[24852]: Active manager daemon y restarted 2024-04-07T20:46:35.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:35 smithi177 ceph-mon[24852]: Activating manager daemon y 2024-04-07T20:46:35.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:35 smithi177 ceph-mon[24852]: osdmap e2: 0 total, 0 up, 0 in 2024-04-07T20:46:36.203 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:36.203 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-07T20:46:36.203 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "initialized": true 2024-04-07T20:46:36.203 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:36.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:46:36] ENGINE Bus STARTING 2024-04-07T20:46:36.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:46:36] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:46:36] ENGINE Bus STARTED 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: mgrmap e6: y(active, starting, since 0.0526331s) 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: Manager daemon y is now available 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:36.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:36.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:36.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:46:36.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:36 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:46:36.707 INFO:teuthology.orchestra.run.smithi177.stderr:mgr epoch 5 is available 2024-04-07T20:46:36.707 INFO:teuthology.orchestra.run.smithi177.stderr:Setting orchestrator backend to cephadm... 2024-04-07T20:46:37.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: [07/Apr/2024:20:46:36] ENGINE Bus STARTING 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: mgrmap e7: y(active, since 1.0609s) 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: [07/Apr/2024:20:46:36] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: [07/Apr/2024:20:46:36] ENGINE Bus STARTED 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:37.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:37 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:38.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:38 smithi177 ceph-mon[24852]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:38.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:38 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:38.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:38 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:38.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:38 smithi177 ceph-mon[24852]: mgrmap e8: y(active, since 2s) 2024-04-07T20:46:38.453 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: value unchanged 2024-04-07T20:46:39.069 INFO:teuthology.orchestra.run.smithi177.stderr:Generating ssh key... 2024-04-07T20:46:39.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-mon[24852]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:40.223 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: Generating public/private rsa key pair. 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: Your identification has been saved in /tmp/tmpsy3syac8/key. 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: Your public key has been saved in /tmp/tmpsy3syac8/key.pub. 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: The key fingerprint is: 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: SHA256:axYap8oA4O45m1sdvCHcpUfaet4eyOBwRzAPG34GZnA ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: The key's randomart image is: 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: +---[RSA 3072]----+ 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: | ..E | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: | = O | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: |. o B | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: |o . o X | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: |.. + X.+S | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: |.. * X=.o | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: | ... =o++. | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: |..+o .oo. . | 2024-04-07T20:46:40.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: | *+ o ..o | 2024-04-07T20:46:40.225 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: +----[SHA256]-----+ 2024-04-07T20:46:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/gYUoFxyMzldr2TOXs39kUp5O3t+wNq/2kQlJ9a2n/zBxooJSH1LoH00MgkDYxIwROxcxTlgfey9Ayajq16qBcLCpkq8wEz1vP4Pjc/qM2wHNpak2YK4siGo08ZyoywFZK019k7TuehTKhMPKciKNzTj1opdNTU0LxOjjiifvrxsUe9lUT6Z38HZjnQpIOLpEjMNMG7UqdP6/S/34Sy2AQJue2SO++6ZdSwqyojHy9z+GdUSbor5H7Eoux5zRBadZbOQ8bTO+5npYll3KwL1/JqwyXBOCCoaXU5qMoEZGvTHhOunAzsPnfKeA5remmazZi4CfPs/QgiXWFYptGJuX0kmzDJE1/Uf0D9s+pFYFRJ96XILZ6pzxoZlTILIerOffho8io7jMn/ouslK1RO+GLmlcydlnAJHnvkHusDLzLco+i1IOQNRaiU7SB3lF493bWetlWHjq20yjcqHO7hPHDrUVULNUDSmKxRGKFNtxTS1EHVhpZoe8Q+4hTgNCUU= ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:46:41.077 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:40 smithi177 ceph-mon[24852]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:41.078 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:40 smithi177 ceph-mon[24852]: Generating ssh key... 2024-04-07T20:46:41.078 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:40 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:41.078 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:40 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:41.366 INFO:teuthology.orchestra.run.smithi177.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-07T20:46:41.366 INFO:teuthology.orchestra.run.smithi177.stderr:Adding key to root@localhost authorized_keys... 2024-04-07T20:46:41.366 INFO:teuthology.orchestra.run.smithi177.stderr:Adding host smithi177... 2024-04-07T20:46:41.934 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:41 smithi177 ceph-mon[24852]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:42.903 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: Added host 'smithi177' with addr '172.21.15.177' 2024-04-07T20:46:43.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:43 smithi177 ceph-mon[24852]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "addr": "172.21.15.177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:43.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:43 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:43.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:43 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:43.589 INFO:teuthology.orchestra.run.smithi177.stderr:Deploying unmanaged mon service... 2024-04-07T20:46:44.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:44 smithi177 ceph-mon[24852]: Deploying cephadm binary to smithi177 2024-04-07T20:46:44.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:44 smithi177 ceph-mon[24852]: Added host smithi177 2024-04-07T20:46:44.258 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-07T20:46:44.823 INFO:teuthology.orchestra.run.smithi177.stderr:Deploying unmanaged mgr service... 2024-04-07T20:46:45.380 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-07T20:46:45.416 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:45 smithi177 ceph-mon[24852]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:45.416 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:45 smithi177 ceph-mon[24852]: Saving service mon spec with placement count:5 2024-04-07T20:46:45.416 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:45 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:46.649 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:46 smithi177 ceph-mon[24852]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:46:46.650 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:46 smithi177 ceph-mon[24852]: Saving service mgr spec with placement count:2 2024-04-07T20:46:46.650 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:46 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:47.682 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:47 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:47.682 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:47 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:47.682 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:47 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4053234892' entity='client.admin' 2024-04-07T20:46:49.207 INFO:teuthology.orchestra.run.smithi177.stderr:Enabling the dashboard module... 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/308240190' entity='client.admin' 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-04-07T20:46:49.474 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:49 smithi177 ceph-mon[24852]: from='mgr.14120 172.21.15.177:0/2283183135' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]': finished 2024-04-07T20:46:50.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:50 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/140654624' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-07T20:46:50.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setuser ceph since I am not root 2024-04-07T20:46:50.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setgroup ceph since I am not root 2024-04-07T20:46:50.831 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:50.741+0000 7f8dc23af000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:46:50.831 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:50.831+0000 7f8dc23af000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:46:51.439 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/140654624' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-07T20:46:51.439 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-mon[24852]: mgrmap e9: y(active, since 15s) 2024-04-07T20:46:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-07T20:46:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "available": true, 2024-04-07T20:46:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-07T20:46:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-07T20:46:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:51.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:51.439+0000 7f8dc23af000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:46:51.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:51.530+0000 7f8dc23af000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:46:51.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:51.618+0000 7f8dc23af000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:46:52.047 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:51.771+0000 7f8dc23af000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:46:52.047 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:51.851+0000 7f8dc23af000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:46:52.047 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:51.962+0000 7f8dc23af000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:46:52.047 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:52.047+0000 7f8dc23af000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:46:52.171 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for the mgr to restart... 2024-04-07T20:46:52.171 INFO:teuthology.orchestra.run.smithi177.stderr:Waiting for mgr epoch 9... 2024-04-07T20:46:52.303 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:52 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3496344880' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-07T20:46:52.303 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:52.217+0000 7f8dc23af000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:46:52.303 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:52.304+0000 7f8dc23af000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:46:53.659 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:53.339+0000 7f8dc23af000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:46:53.659 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:53.469+0000 7f8dc23af000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:46:53.659 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:53.659+0000 7f8dc23af000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:46:54.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:53.746+0000 7f8dc23af000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:46:54.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:53.881+0000 7f8dc23af000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:46:54.506 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:54 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:54.202+0000 7f8dc23af000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:46:54.506 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:54 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:54.422+0000 7f8dc23af000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:46:54.506 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:54 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:54.506+0000 7f8dc23af000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:46:54.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:54 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:54.675+0000 7f8dc23af000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:46:55.881 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:55 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:55.590+0000 7f8dc23af000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:46:55.881 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:55 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:55.882+0000 7f8dc23af000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:46:56.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:55 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:55.976+0000 7f8dc23af000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:46:56.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:56 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:46:56.126+0000 7f8dc23af000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:46:56.907 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:56 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:46:56] ENGINE Bus STARTING 2024-04-07T20:46:56.907 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:56 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:46:56] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:46:56.907 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:46:56 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:46:56] ENGINE Bus STARTED 2024-04-07T20:46:57.203 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: Active manager daemon y restarted 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: Activating manager daemon y 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: osdmap e3: 0 total, 0 up, 0 in 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: mgrmap e10: y(active, starting, since 0.0552311s) 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: Manager daemon y is now available 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:46:57.204 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:46:57.205 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:46:57.205 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: { 2024-04-07T20:46:57.205 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-07T20:46:57.205 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: "initialized": true 2024-04-07T20:46:57.205 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: } 2024-04-07T20:46:57.939 INFO:teuthology.orchestra.run.smithi177.stderr:mgr epoch 9 is available 2024-04-07T20:46:57.939 INFO:teuthology.orchestra.run.smithi177.stderr:Generating a dashboard self-signed certificate... 2024-04-07T20:46:58.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:58 smithi177 ceph-mon[24852]: [07/Apr/2024:20:46:56] ENGINE Bus STARTING 2024-04-07T20:46:58.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:58 smithi177 ceph-mon[24852]: [07/Apr/2024:20:46:56] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:46:58.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:58 smithi177 ceph-mon[24852]: [07/Apr/2024:20:46:56] ENGINE Bus STARTED 2024-04-07T20:46:58.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:58 smithi177 ceph-mon[24852]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-07T20:46:58.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:58 smithi177 ceph-mon[24852]: mgrmap e11: y(active, since 1.05803s) 2024-04-07T20:46:58.467 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:58 smithi177 ceph-mon[24852]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-07T20:46:58.665 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-07T20:46:59.305 INFO:teuthology.orchestra.run.smithi177.stderr:Creating initial admin user... 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: mgrmap e12: y(active, since 2s) 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi177", "caps": []}]: dispatch 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:46:59.427 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:46:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:00.132 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$2eMdxUJxpwVJmHZWtCQ8SeINS2fqJqiXxkse.nk0OYUJhpWDvOc6G", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712522820, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-07T20:47:00.424 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:00 smithi177 ceph-mon[24852]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:00.424 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:00 smithi177 ceph-mon[24852]: Deploying daemon agent.smithi177 on smithi177 2024-04-07T20:47:00.424 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:00.727 INFO:teuthology.orchestra.run.smithi177.stderr:Fetching dashboard port number... 2024-04-07T20:47:01.352 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:01 smithi177 ceph-mon[24852]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:01.353 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: 8443 2024-04-07T20:47:02.009 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: disabled 2024-04-07T20:47:02.016 INFO:teuthology.orchestra.run.smithi177.stderr:systemctl: inactive 2024-04-07T20:47:02.017 INFO:teuthology.orchestra.run.smithi177.stderr:firewalld.service is not enabled 2024-04-07T20:47:02.017 INFO:teuthology.orchestra.run.smithi177.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr:Ceph Dashboard is now available at: 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr: URL: https://smithi177.front.sepia.ceph.com:8443/ 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr: User: admin 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr: Password: 52n70wba5c 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:02.019 INFO:teuthology.orchestra.run.smithi177.stderr:Enabling autotune for osd_memory_target 2024-04-07T20:47:02.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:02 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3100024450' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-07T20:47:04.236 INFO:teuthology.orchestra.run.smithi177.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-07T20:47:04.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:04 smithi177 ceph-mon[24852]: mgrmap e13: y(active, since 6s) 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr:You can access the Ceph CLI with: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: ceph telemetry on 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr:For more information see: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T20:47:04.831 INFO:teuthology.orchestra.run.smithi177.stderr:Bootstrap complete. 2024-04-07T20:47:05.020 INFO:tasks.cephadm:Fetching config... 2024-04-07T20:47:05.020 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:05.020 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-07T20:47:05.035 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-07T20:47:05.036 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:05.036 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-07T20:47:05.091 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-07T20:47:05.091 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:05.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-07T20:47:05.202 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-07T20:47:05.202 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:05.202 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-07T20:47:05.258 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-07T20:47:05.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/gYUoFxyMzldr2TOXs39kUp5O3t+wNq/2kQlJ9a2n/zBxooJSH1LoH00MgkDYxIwROxcxTlgfey9Ayajq16qBcLCpkq8wEz1vP4Pjc/qM2wHNpak2YK4siGo08ZyoywFZK019k7TuehTKhMPKciKNzTj1opdNTU0LxOjjiifvrxsUe9lUT6Z38HZjnQpIOLpEjMNMG7UqdP6/S/34Sy2AQJue2SO++6ZdSwqyojHy9z+GdUSbor5H7Eoux5zRBadZbOQ8bTO+5npYll3KwL1/JqwyXBOCCoaXU5qMoEZGvTHhOunAzsPnfKeA5remmazZi4CfPs/QgiXWFYptGJuX0kmzDJE1/Uf0D9s+pFYFRJ96XILZ6pzxoZlTILIerOffho8io7jMn/ouslK1RO+GLmlcydlnAJHnvkHusDLzLco+i1IOQNRaiU7SB3lF493bWetlWHjq20yjcqHO7hPHDrUVULNUDSmKxRGKFNtxTS1EHVhpZoe8Q+4hTgNCUU= ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-07T20:47:05.324 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:05 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/753863771' entity='client.admin' 2024-04-07T20:47:05.450 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/gYUoFxyMzldr2TOXs39kUp5O3t+wNq/2kQlJ9a2n/zBxooJSH1LoH00MgkDYxIwROxcxTlgfey9Ayajq16qBcLCpkq8wEz1vP4Pjc/qM2wHNpak2YK4siGo08ZyoywFZK019k7TuehTKhMPKciKNzTj1opdNTU0LxOjjiifvrxsUe9lUT6Z38HZjnQpIOLpEjMNMG7UqdP6/S/34Sy2AQJue2SO++6ZdSwqyojHy9z+GdUSbor5H7Eoux5zRBadZbOQ8bTO+5npYll3KwL1/JqwyXBOCCoaXU5qMoEZGvTHhOunAzsPnfKeA5remmazZi4CfPs/QgiXWFYptGJuX0kmzDJE1/Uf0D9s+pFYFRJ96XILZ6pzxoZlTILIerOffho8io7jMn/ouslK1RO+GLmlcydlnAJHnvkHusDLzLco+i1IOQNRaiU7SB3lF493bWetlWHjq20yjcqHO7hPHDrUVULNUDSmKxRGKFNtxTS1EHVhpZoe8Q+4hTgNCUU= ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:47:05.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/gYUoFxyMzldr2TOXs39kUp5O3t+wNq/2kQlJ9a2n/zBxooJSH1LoH00MgkDYxIwROxcxTlgfey9Ayajq16qBcLCpkq8wEz1vP4Pjc/qM2wHNpak2YK4siGo08ZyoywFZK019k7TuehTKhMPKciKNzTj1opdNTU0LxOjjiifvrxsUe9lUT6Z38HZjnQpIOLpEjMNMG7UqdP6/S/34Sy2AQJue2SO++6ZdSwqyojHy9z+GdUSbor5H7Eoux5zRBadZbOQ8bTO+5npYll3KwL1/JqwyXBOCCoaXU5qMoEZGvTHhOunAzsPnfKeA5remmazZi4CfPs/QgiXWFYptGJuX0kmzDJE1/Uf0D9s+pFYFRJ96XILZ6pzxoZlTILIerOffho8io7jMn/ouslK1RO+GLmlcydlnAJHnvkHusDLzLco+i1IOQNRaiU7SB3lF493bWetlWHjq20yjcqHO7hPHDrUVULNUDSmKxRGKFNtxTS1EHVhpZoe8Q+4hTgNCUU= ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-07T20:47:05.673 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/gYUoFxyMzldr2TOXs39kUp5O3t+wNq/2kQlJ9a2n/zBxooJSH1LoH00MgkDYxIwROxcxTlgfey9Ayajq16qBcLCpkq8wEz1vP4Pjc/qM2wHNpak2YK4siGo08ZyoywFZK019k7TuehTKhMPKciKNzTj1opdNTU0LxOjjiifvrxsUe9lUT6Z38HZjnQpIOLpEjMNMG7UqdP6/S/34Sy2AQJue2SO++6ZdSwqyojHy9z+GdUSbor5H7Eoux5zRBadZbOQ8bTO+5npYll3KwL1/JqwyXBOCCoaXU5qMoEZGvTHhOunAzsPnfKeA5remmazZi4CfPs/QgiXWFYptGJuX0kmzDJE1/Uf0D9s+pFYFRJ96XILZ6pzxoZlTILIerOffho8io7jMn/ouslK1RO+GLmlcydlnAJHnvkHusDLzLco+i1IOQNRaiU7SB3lF493bWetlWHjq20yjcqHO7hPHDrUVULNUDSmKxRGKFNtxTS1EHVhpZoe8Q+4hTgNCUU= ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:47:05.687 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-07T20:47:06.157 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:05 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:47:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:10.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:47:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:10.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:10 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3162017462' entity='client.admin' 2024-04-07T20:47:10.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:10.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:10.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:10.605 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-07T20:47:10.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-07T20:47:10.916 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:47:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:11 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:11 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:13.160 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2024-04-07T20:47:13.161 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:47:13.161 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2024-04-07T20:47:13.179 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:47:13.179 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:47:13.236 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2024-04-07T20:47:13.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi187 2024-04-07T20:47:13.538 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:13.538 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:13.538 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:13.538 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:13.539 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:13.539 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-07T20:47:13.539 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:47:13.539 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:13.539 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:15.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:15 smithi177 ceph-mon[24852]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:16.304 INFO:teuthology.orchestra.run.smithi177.stdout:Added host 'smithi187' with addr '172.21.15.187' 2024-04-07T20:47:16.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: Deploying cephadm binary to smithi187 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:17.494 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:47:17.495 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:17.495 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi187", "caps": []}]: dispatch 2024-04-07T20:47:17.495 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi187", "caps": []}]': finished 2024-04-07T20:47:18.375 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:18 smithi177 ceph-mon[24852]: Added host smithi187 2024-04-07T20:47:18.375 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:18 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:18.375 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:18 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:18.375 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:18 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:18.375 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:18 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:18.555 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:47:18.555 INFO:teuthology.orchestra.run.smithi177.stdout:[{"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}, {"addr": "172.21.15.187", "hostname": "smithi187", "labels": [], "status": ""}] 2024-04-07T20:47:19.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:19.110 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-07T20:47:19.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-07T20:47:19.412 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: Deploying daemon agent.smithi187 on smithi187 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: Updating smithi187:/etc/ceph/ceph.conf 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:19.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:19.708 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:19 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:19] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:20.384 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:20 smithi177 ceph-mon[24852]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:47:20.385 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:20 smithi177 ceph-mon[24852]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:47:20.385 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:20.385 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:21.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:20 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:20] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:21.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:21 smithi177 ceph-mon[24852]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:21.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:21.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:21.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:21 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2103079086' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-07T20:47:21.980 INFO:teuthology.orchestra.run.smithi177.stderr:adjusted tunables profile to default 2024-04-07T20:47:22.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:22.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:22 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2103079086' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-07T20:47:22.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:22 smithi177 ceph-mon[24852]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:22.561 INFO:tasks.cephadm:Adding mon.a on smithi177 2024-04-07T20:47:22.561 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-04-07T20:47:22.561 INFO:tasks.cephadm:Adding mon.b on smithi187 2024-04-07T20:47:22.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '3;smithi177:172.21.15.177=a;smithi177:[v2:172.21.15.177:3301,v1:172.21.15.177:6790]=c;smithi187:172.21.15.187=b' 2024-04-07T20:47:23.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:23 smithi177 ceph-mon[24852]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:24.483 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2024-04-07T20:47:25.113 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.c.service 2024-04-07T20:47:25.115 DEBUG:teuthology.orchestra.run.smithi187:mon.b> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.b.service 2024-04-07T20:47:25.118 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-07T20:47:25.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T20:47:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:25.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:25.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:25.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:25.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:26.492 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:26 smithi177 ceph-mon[24852]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi177:172.21.15.177=a;smithi177:[v2:172.21.15.177:3301,v1:172.21.15.177:6790]=c;smithi187:172.21.15.187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:26.493 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:26 smithi177 ceph-mon[24852]: Saving service mon spec with placement smithi177:172.21.15.177=a;smithi177:[v2:172.21.15.177:3301,v1:172.21.15.177:6790]=c;smithi187:172.21.15.187=b;count:3 2024-04-07T20:47:26.493 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:26 smithi177 ceph-mon[24852]: Deploying daemon mon.c on smithi177 2024-04-07T20:47:27.177 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:47:27.177 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","modified":"2024-04-07T20:46:00.238331Z","created":"2024-04-07T20:46:00.238331Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T20:47:27.179 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2024-04-07T20:47:27.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:27 smithi177 ceph-mon[24852]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:27.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:27 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/885350734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:28.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:28 smithi177 ceph-mon[29851]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-07T20:47:28.780 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-07T20:47:28.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T20:47:29.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:28 smithi177 bash[29785]: 5c620a76fdbb30d25bcecd0c71399bbd0a0a066f00940a88c103d455e60745c0 2024-04-07T20:47:29.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:28 smithi177 systemd[1]: Started Ceph mon.c for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:47:34.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:47:33] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: mon.a calling monitor election 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: mon.c calling monitor election 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: monmap e2: 2 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: fsmap 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: mgrmap e13: y(active, since 37s) 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: overall HEALTH_OK 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:34.261 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: mon.a calling monitor election 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: mon.c calling monitor election 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-07T20:47:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: monmap e2: 2 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: fsmap 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: mgrmap e13: y(active, since 37s) 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: overall HEALTH_OK 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:34.263 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:34.263 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:47:34.264 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","modified":"2024-04-07T20:47:28.726615Z","created":"2024-04-07T20:46:00.238331Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3301","nonce":0},{"type":"v1","addr":"172.21.15.177:6790","nonce":0}]},"addr":"172.21.15.177:6790/0","public_addr":"172.21.15.177:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-07T20:47:34.264 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-04-07T20:47:35.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[24852]: Deploying daemon mon.b on smithi187 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[24852]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/2264604022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[29851]: Deploying daemon mon.b on smithi187 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[29851]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/2264604022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:35.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:34 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:37.213 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-07T20:47:37.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T20:47:37.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:37 smithi177 ceph-mon[24852]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:37.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:37 smithi177 ceph-mon[29851]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:39.124 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:47:39.124 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","modified":"2024-04-07T20:47:28.726615Z","created":"2024-04-07T20:46:00.238331Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3301","nonce":0},{"type":"v1","addr":"172.21.15.177:6790","nonce":0}]},"addr":"172.21.15.177:6790/0","public_addr":"172.21.15.177:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-07T20:47:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-04-07T20:47:39.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:39 smithi177 ceph-mon[24852]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:39.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:39 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/2010368091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:39 smithi177 ceph-mon[29851]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:39 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/2010368091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:40.667 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-07T20:47:40.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T20:47:46.171 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:45] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:46.171 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:46 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: mon.a calling monitor election 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: mon.c calling monitor election 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: monmap e3: 3 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],b=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: fsmap 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: mgrmap e13: y(active, since 49s) 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: mon.b (rank 2) addr [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] is down (out of quorum) 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:46.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: mon.a calling monitor election 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: mon.c calling monitor election 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.174 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: monmap e3: 3 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],b=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: fsmap 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: mgrmap e13: y(active, since 49s) 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: mon.b (rank 2) addr [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] is down (out of quorum) 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:46.175 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:46.176 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:46.176 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:46.559 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-07T20:47:46.559 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":3,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","modified":"2024-04-07T20:47:40.879696Z","created":"2024-04-07T20:46:00.238331Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3301","nonce":0},{"type":"v1","addr":"172.21.15.177:6790","nonce":0}]},"addr":"172.21.15.177:6790/0","public_addr":"172.21.15.177:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-07T20:47:46.560 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 3 2024-04-07T20:47:47.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-07T20:47:47.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: Updating smithi187:/etc/ceph/ceph.conf 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: Reconfiguring mon.a (unknown last config time)... 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: Reconfiguring daemon mon.a on smithi177 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/633656214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:47.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: Updating smithi187:/etc/ceph/ceph.conf 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: Reconfiguring mon.a (unknown last config time)... 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:47.145 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: Reconfiguring daemon mon.a on smithi177 2024-04-07T20:47:47.146 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:47.146 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:47.146 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/633656214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T20:47:47.146 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:47.247 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-07T20:47:47.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-07T20:47:47.454 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:47 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:47.911 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:47 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: mon.b calling monitor election 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: mon.b calling monitor election 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: mon.a calling monitor election 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: mon.c calling monitor election 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: monmap e3: 3 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],b=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: fsmap 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: mgrmap e13: y(active, since 51s) 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: Cluster is now healthy 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: overall HEALTH_OK 2024-04-07T20:47:48.601 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[29851]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: mon.b calling monitor election 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: mon.b calling monitor election 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: mon.a calling monitor election 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: mon.c calling monitor election 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: monmap e3: 3 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],b=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: fsmap 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: mgrmap e13: y(active, since 51s) 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: Cluster is now healthy 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: overall HEALTH_OK 2024-04-07T20:47:48.602 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-mon[24852]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:48.684 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: mon.b calling monitor election 2024-04-07T20:47:48.684 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: mon.b calling monitor election 2024-04-07T20:47:48.684 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: mon.a calling monitor election 2024-04-07T20:47:48.684 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: mon.c calling monitor election 2024-04-07T20:47:48.684 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-07T20:47:48.684 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: monmap e3: 3 mons at {a=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],b=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0],c=[v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0]} 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: fsmap 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: mgrmap e13: y(active, since 51s) 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: Cluster is now healthy 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: overall HEALTH_OK 2024-04-07T20:47:48.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:48 smithi187 ceph-mon[25907]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:48.879 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:47:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:49.501 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:47:49] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:49.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: Reconfiguring mon.c (monmap changed)... 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: Reconfiguring daemon mon.c on smithi177 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: Reconfiguring mon.c (monmap changed)... 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: Reconfiguring daemon mon.c on smithi177 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:49.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:50.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:50.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: Reconfiguring mon.c (monmap changed)... 2024-04-07T20:47:50.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: Reconfiguring daemon mon.c on smithi177 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:50.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:50.756 INFO:teuthology.orchestra.run.smithi177.stdout:# minimal ceph.conf for c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:47:50.757 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-04-07T20:47:50.757 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = c7f93d4c-f51f-11ee-b649-cb9ed24678a4 2024-04-07T20:47:50.757 INFO:teuthology.orchestra.run.smithi177.stdout: mon_host = [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] 2024-04-07T20:47:50.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:50 smithi177 ceph-mon[29851]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:50.757 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:50 smithi177 ceph-mon[24852]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:51.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:50 smithi187 ceph-mon[25907]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:51.336 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:47:51] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:47:51.552 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-07T20:47:51.552 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:51.553 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-04-07T20:47:51.582 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:51.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:47:51.650 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:47:51.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2024-04-07T20:47:51.686 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:47:51.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:47:51.757 INFO:tasks.cephadm:Adding mgr.y on smithi177 2024-04-07T20:47:51.757 INFO:tasks.cephadm:Adding mgr.x on smithi187 2024-04-07T20:47:51.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch apply mgr '2;smithi177=y;smithi187=x' 2024-04-07T20:47:51.891 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3406454844' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:51.892 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:51.892 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:51.892 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3406454844' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:51.892 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:51.892 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:51 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:52.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:51 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3406454844' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:52.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:51 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:52.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:51 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:53.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:53 smithi187 ceph-mon[25907]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:53.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:53 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:53.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:53 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:53.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:53 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:53.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:53 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:53.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[24852]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:53.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[29851]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T20:47:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:53 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:54.393 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:54 smithi187 ceph-mon[25907]: Reconfiguring mon.b (monmap changed)... 2024-04-07T20:47:54.393 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:54 smithi187 ceph-mon[25907]: Reconfiguring daemon mon.b on smithi187 2024-04-07T20:47:54.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:54 smithi177 ceph-mon[24852]: Reconfiguring mon.b (monmap changed)... 2024-04-07T20:47:54.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:54 smithi177 ceph-mon[24852]: Reconfiguring daemon mon.b on smithi187 2024-04-07T20:47:54.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:54 smithi177 ceph-mon[29851]: Reconfiguring mon.b (monmap changed)... 2024-04-07T20:47:54.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:54 smithi177 ceph-mon[29851]: Reconfiguring daemon mon.b on smithi187 2024-04-07T20:47:54.728 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mgr update... 2024-04-07T20:47:55.328 DEBUG:teuthology.orchestra.run.smithi187:mgr.x> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.x.service 2024-04-07T20:47:55.330 INFO:tasks.cephadm:Deploying OSDs... 2024-04-07T20:47:55.330 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:47:55.330 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-07T20:47:55.350 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-07T20:47:55.350 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-07T20:47:55.407 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-07T20:47:55.407 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:55.407 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-07T20:47:55.407 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:55.407 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:55.408 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-07 20:47:50.079914758 +0000 2024-04-07T20:47:55.408 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-07 20:44:51.492867572 +0000 2024-04-07T20:47:55.408 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-07 20:44:51.492867572 +0000 2024-04-07T20:47:55.408 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-07 20:44:51.492867572 +0000 2024-04-07T20:47:55.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-07T20:47:55.477 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-07T20:47:55.477 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-07T20:47:55.477 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.00025377 s, 2.0 MB/s 2024-04-07T20:47:55.478 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-07T20:47:55.535 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-07T20:47:55.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:55.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:47:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.590 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-07T20:47:55.590 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:55.590 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-07T20:47:55.590 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:55.590 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:55.591 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-07 20:47:50.079914758 +0000 2024-04-07T20:47:55.591 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-07 20:44:51.841858076 +0000 2024-04-07T20:47:55.591 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-07 20:44:51.841858076 +0000 2024-04-07T20:47:55.591 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-07 20:44:51.841858076 +0000 2024-04-07T20:47:55.591 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-07T20:47:55.655 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:55.655 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.655 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:55.655 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:55.656 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:47:55.657 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:55.658 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-07T20:47:55.659 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-07T20:47:55.659 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000248552 s, 2.1 MB/s 2024-04-07T20:47:55.659 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-07T20:47:55.716 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-07 20:47:50.079914758 +0000 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-07 20:44:52.182848797 +0000 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-07 20:44:52.182848797 +0000 2024-04-07T20:47:55.772 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-07 20:44:52.182848797 +0000 2024-04-07T20:47:55.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-07T20:47:55.840 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-07T20:47:55.840 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-07T20:47:55.840 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000239022 s, 2.1 MB/s 2024-04-07T20:47:55.841 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-07T20:47:55.898 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-07T20:47:55.952 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-07T20:47:55.952 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:55.952 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-07T20:47:55.953 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:55.953 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:55.953 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-07 20:47:50.079914758 +0000 2024-04-07T20:47:55.953 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-07 20:44:52.523839518 +0000 2024-04-07T20:47:55.953 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-07 20:44:52.523839518 +0000 2024-04-07T20:47:55.953 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-07 20:44:52.523839518 +0000 2024-04-07T20:47:55.953 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-07T20:47:56.018 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-07T20:47:56.018 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-07T20:47:56.018 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000190201 s, 2.7 MB/s 2024-04-07T20:47:56.019 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-07T20:47:56.075 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:47:56.075 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-04-07T20:47:56.091 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-07T20:47:56.091 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-04-07T20:47:56.145 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-07T20:47:56.145 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-07 20:47:48.653126904 +0000 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-07 20:44:42.426862612 +0000 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-07 20:44:42.426862612 +0000 2024-04-07T20:47:56.146 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-07 20:44:42.426862612 +0000 2024-04-07T20:47:56.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-07T20:47:56.212 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-07T20:47:56.212 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-07T20:47:56.212 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000183991 s, 2.8 MB/s 2024-04-07T20:47:56.213 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-07T20:47:56.269 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-07 20:47:48.653126904 +0000 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-07 20:44:42.775856896 +0000 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-07 20:44:42.775856896 +0000 2024-04-07T20:47:56.323 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-07 20:44:42.775856896 +0000 2024-04-07T20:47:56.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-07T20:47:56.388 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-07T20:47:56.388 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-07T20:47:56.388 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000197024 s, 2.6 MB/s 2024-04-07T20:47:56.389 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-07T20:47:56.444 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-04-07T20:47:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-07T20:47:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-07 20:47:48.653126904 +0000 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-07 20:44:43.105851492 +0000 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-07 20:44:43.105851492 +0000 2024-04-07T20:47:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-07 20:44:43.105851492 +0000 2024-04-07T20:47:56.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-07T20:47:56.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:56 smithi187 ceph-mon[25907]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi177=y;smithi187=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:56.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:56 smithi187 ceph-mon[25907]: Saving service mgr spec with placement smithi177=y;smithi187=x;count:2 2024-04-07T20:47:56.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:56 smithi187 ceph-mon[25907]: Deploying daemon mgr.x on smithi187 2024-04-07T20:47:56.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:56 smithi187 ceph-mon[25907]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:56.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:56 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:47:56.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:56 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:47:56.564 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-07T20:47:56.564 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-07T20:47:56.564 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000189083 s, 2.7 MB/s 2024-04-07T20:47:56.565 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-07T20:47:56.623 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-04-07T20:47:56.679 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-07T20:47:56.679 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T20:47:56.679 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-07T20:47:56.679 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T20:47:56.679 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T20:47:56.680 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-07 20:47:48.653126904 +0000 2024-04-07T20:47:56.680 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-07 20:44:43.437846054 +0000 2024-04-07T20:47:56.680 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-07 20:44:43.437846054 +0000 2024-04-07T20:47:56.680 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-07 20:44:43.437846054 +0000 2024-04-07T20:47:56.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-07T20:47:56.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[29851]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi177=y;smithi187=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:56.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[29851]: Saving service mgr spec with placement smithi177=y;smithi187=x;count:2 2024-04-07T20:47:56.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[29851]: Deploying daemon mgr.x on smithi187 2024-04-07T20:47:56.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[29851]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[24852]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi177=y;smithi187=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[24852]: Saving service mgr spec with placement smithi177=y;smithi187=x;count:2 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[24852]: Deploying daemon mgr.x on smithi187 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[24852]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:47:56.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:56 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:47:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-07T20:47:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-07T20:47:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000208674 s, 2.5 MB/s 2024-04-07T20:47:56.764 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-07T20:47:56.823 INFO:tasks.cephadm:Deploying osd.0 on smithi177 with /dev/vg_nvme/lv_4... 2024-04-07T20:47:56.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-07T20:47:57.170 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:56 smithi187 systemd[1]: Starting Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:47:57.465 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 podman[27308]: 2024-04-07 20:47:57.074784868 +0000 UTC m=+0.018574622 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:47:57.465 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 podman[27308]: 2024-04-07 20:47:57.172781236 +0000 UTC m=+0.116571036 container create a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, distribution-scope=public, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T20:47:57.466 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 podman[27308]: 2024-04-07 20:47:57.366545363 +0000 UTC m=+0.310335117 container init a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, name=centos-stream, release=754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T20:47:57.466 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 podman[27308]: 2024-04-07 20:47:57.368982846 +0000 UTC m=+0.312772601 container start a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, version=8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, release=754) 2024-04-07T20:47:57.466 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 ceph-mgr[27327]: -- 172.21.15.187:0/20837708 <== mon.2 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c5e66a340 con 0x556c5e643000 2024-04-07T20:47:57.466 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 bash[27308]: a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 2024-04-07T20:47:57.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 systemd[1]: Started Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:47:58.184 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:57.853+0000 7f943ff52000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:47:58.184 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:57 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:57.942+0000 7f943ff52000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:58.549+0000 7f943ff52000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:58.644+0000 7f943ff52000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:58.727+0000 7f943ff52000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:47:58.728 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: Reconfiguring mgr.y (unknown last config time)... 2024-04-07T20:47:58.729 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:47:58.729 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:47:58.729 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:58.729 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: Reconfiguring daemon mgr.y on smithi177 2024-04-07T20:47:58.729 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-mon[25907]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: Reconfiguring mgr.y (unknown last config time)... 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: Reconfiguring daemon mgr.y on smithi177 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[24852]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:58.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: Reconfiguring mgr.y (unknown last config time)... 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: Reconfiguring daemon mgr.y on smithi177 2024-04-07T20:47:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:47:58 smithi177 ceph-mon[29851]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:47:59.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:58.877+0000 7f943ff52000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:47:59.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:58.957+0000 7f943ff52000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:47:59.338 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:59.074+0000 7f943ff52000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:47:59.339 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:59.156+0000 7f943ff52000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:47:59.339 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:59.338+0000 7f943ff52000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:47:59.353 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-07T20:47:59.353 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-07T20:47:59.353 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-07T20:47:59.353 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-07T20:47:59.353 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012735 s, 823 MB/s 2024-04-07T20:47:59.353 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-07T20:47:59.776 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:47:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:47:59.429+0000 7f943ff52000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:48:00.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:48:00.708 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[24852]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:48:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-mon[29851]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:00.797 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:00.534+0000 7f943ff52000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:48:00.797 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:00.664+0000 7f943ff52000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:48:00.797 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:00.797 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:00.798 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:00.798 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:00.798 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:00.798 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:48:00.798 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-mon[25907]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:00.997 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:48:00] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:00.997 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:48:00] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:01.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:00.855+0000 7f943ff52000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:48:01.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:00 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:00.947+0000 7f943ff52000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:48:01.404 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:01.085+0000 7f943ff52000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:48:01.404 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:01.404+0000 7f943ff52000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:48:01.705 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:01.621+0000 7f943ff52000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:48:01.705 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:01.705+0000 7f943ff52000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:48:02.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:01.872+0000 7f943ff52000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:48:02.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:02.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:02.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:02.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:02.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:03.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:02 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:02.777+0000 7f943ff52000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:48:03.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:02 smithi187 ceph-mon[25907]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:02 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:02 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:02 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:02 smithi187 ceph-mon[25907]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:03.252 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[24852]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[24852]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[29851]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:03.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:02 smithi177 ceph-mon[29851]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:03.556 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:03.065+0000 7f943ff52000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:48:03.557 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:03.157+0000 7f943ff52000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:48:03.557 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:48:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:48:03.304+0000 7f943ff52000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:48:04.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[24852]: Standby manager daemon x started 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[29851]: Standby manager daemon x started 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-07T20:48:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:04 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:48:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:04 smithi187 ceph-mon[25907]: Standby manager daemon x started 2024-04-07T20:48:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:04 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-07T20:48:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:04 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:48:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:04 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-07T20:48:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:04 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/990688489' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:48:05.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: mgrmap e14: y(active, since 67s), standbys: x 2024-04-07T20:48:05.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:48:05.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:05.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1220990045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "108f20e3-d403-4cb5-8ef2-e62d0f73fb65"}]: dispatch 2024-04-07T20:48:05.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1220990045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "108f20e3-d403-4cb5-8ef2-e62d0f73fb65"}]': finished 2024-04-07T20:48:05.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: osdmap e5: 1 total, 0 up, 1 in 2024-04-07T20:48:05.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:05.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:05 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2516780433' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:05.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: mgrmap e14: y(active, since 67s), standbys: x 2024-04-07T20:48:05.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:48:05.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:05.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1220990045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "108f20e3-d403-4cb5-8ef2-e62d0f73fb65"}]: dispatch 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1220990045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "108f20e3-d403-4cb5-8ef2-e62d0f73fb65"}]': finished 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: osdmap e5: 1 total, 0 up, 1 in 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2516780433' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: mgrmap e14: y(active, since 67s), standbys: x 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1220990045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "108f20e3-d403-4cb5-8ef2-e62d0f73fb65"}]: dispatch 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1220990045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "108f20e3-d403-4cb5-8ef2-e62d0f73fb65"}]': finished 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: osdmap e5: 1 total, 0 up, 1 in 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:05 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2516780433' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:07.528 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:07 smithi177 ceph-mon[24852]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:07.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:07 smithi177 ceph-mon[29851]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:07.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:07 smithi187 ceph-mon[25907]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:09.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:09 smithi187 ceph-mon[25907]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:09.573 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:09 smithi177 ceph-mon[24852]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:09.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:09 smithi177 ceph-mon[29851]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:10.443 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:48:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:11.223 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:11 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:11.224 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:11 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:11.224 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:11 smithi177 ceph-mon[29851]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:11.224 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:11 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:11.224 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:11 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:11.224 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:11 smithi177 ceph-mon[24852]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:11.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:11 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:11.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:11 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:11.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:11 smithi187 ceph-mon[25907]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:13.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:13 smithi177 ceph-mon[24852]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:13.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-07T20:48:13.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:13.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:13 smithi177 ceph-mon[29851]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:13.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-07T20:48:13.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:13.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:13 smithi187 ceph-mon[25907]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:13.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-07T20:48:13.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:14.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:14 smithi177 ceph-mon[29851]: Deploying daemon osd.0 on smithi177 2024-04-07T20:48:14.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:14 smithi177 ceph-mon[24852]: Deploying daemon osd.0 on smithi177 2024-04-07T20:48:14.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:14 smithi187 ceph-mon[25907]: Deploying daemon osd.0 on smithi177 2024-04-07T20:48:15.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:15 smithi177 ceph-mon[24852]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:15.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:15 smithi177 ceph-mon[29851]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:15.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:15 smithi187 ceph-mon[25907]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[29851]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[24852]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:17.451 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:17.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:17.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:17.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:17.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:17.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:17 smithi187 ceph-mon[25907]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:17.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:17 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:17.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:17 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:17.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:17 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:17.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:17 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:17.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:17 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:19.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:19 smithi187 ceph-mon[25907]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:19.670 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:19 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:48:19] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:19.670 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:19 smithi177 ceph-mon[29851]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:19.671 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:19 smithi177 ceph-mon[24852]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:19.700 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 0 on host 'smithi177' 2024-04-07T20:48:20.290 DEBUG:teuthology.orchestra.run.smithi177:osd.0> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-07T20:48:20.293 INFO:tasks.cephadm:Deploying osd.1 on smithi177 with /dev/vg_nvme/lv_3... 2024-04-07T20:48:20.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-07T20:48:20.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[29851]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T20:48:20.317 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T20:48:20.318 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:20 smithi177 ceph-mon[24852]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:20.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T20:48:20.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:20 smithi187 ceph-mon[25907]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T20:48:21.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi177 to 15040M 2024-04-07T20:48:21.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:21.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: osdmap e6: 1 total, 0 up, 1 in 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: from='osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[24852]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi177 to 15040M 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: osdmap e6: 1 total, 0 up, 1 in 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:21.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: from='osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:21.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-mon[29851]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi177 to 15040M 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: osdmap e6: 1 total, 0 up, 1 in 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: from='osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:21 smithi187 ceph-mon[25907]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:21.782 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0[33080]: 2024-04-07T20:48:21.764+0000 7f4b87607700 -1 osd.0 0 waiting for initial osdmap 2024-04-07T20:48:21.782 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:48:21 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0[33080]: 2024-04-07T20:48:21.767+0000 7f4b7ff9a700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:48:22.242 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-07T20:48:22.242 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-07T20:48:22.242 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-07T20:48:22.242 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-07T20:48:22.242 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123885 s, 846 MB/s 2024-04-07T20:48:22.242 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-07T20:48:22.833 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-04-07T20:48:23.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[29851]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:48:23.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[29851]: osdmap e7: 1 total, 0 up, 1 in 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[29851]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[24852]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[24852]: osdmap e7: 1 total, 0 up, 1 in 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:23.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:22 smithi177 ceph-mon[24852]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:23.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:22 smithi187 ceph-mon[25907]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:48:23.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:22 smithi187 ceph-mon[25907]: osdmap e7: 1 total, 0 up, 1 in 2024-04-07T20:48:23.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:22 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:23.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:22 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:23.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:22 smithi187 ceph-mon[25907]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T20:48:24.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[24852]: osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055] boot 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[24852]: osdmap e8: 1 total, 1 up, 1 in 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[29851]: osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055] boot 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[29851]: osdmap e8: 1 total, 1 up, 1 in 2024-04-07T20:48:24.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:24.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:23 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:48:24.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:23 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:48:24.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:23 smithi187 ceph-mon[25907]: osd.0 [v2:172.21.15.177:6802/3585323055,v1:172.21.15.177:6803/3585323055] boot 2024-04-07T20:48:24.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:23 smithi187 ceph-mon[25907]: osdmap e8: 1 total, 1 up, 1 in 2024-04-07T20:48:24.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:48:25.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[29851]: osdmap e9: 1 total, 1 up, 1 in 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[29851]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[24852]: osdmap e9: 1 total, 1 up, 1 in 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[24852]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:25 smithi187 ceph-mon[25907]: osdmap e9: 1 total, 1 up, 1 in 2024-04-07T20:48:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:25 smithi187 ceph-mon[25907]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:25 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:25 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:25 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:26.497 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:26 smithi177 ceph-mon[24852]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:26.498 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:26 smithi177 ceph-mon[29851]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:26 smithi187 ceph-mon[25907]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:27.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[29851]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2736323480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]: dispatch 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]: dispatch 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]': finished 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[29851]: osdmap e10: 2 total, 1 up, 2 in 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[24852]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2736323480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]: dispatch 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]: dispatch 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]': finished 2024-04-07T20:48:27.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[24852]: osdmap e10: 2 total, 1 up, 2 in 2024-04-07T20:48:27.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:27 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:27 smithi187 ceph-mon[25907]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:27 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2736323480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]: dispatch 2024-04-07T20:48:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:27 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]: dispatch 2024-04-07T20:48:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:27 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694"}]': finished 2024-04-07T20:48:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:27 smithi187 ceph-mon[25907]: osdmap e10: 2 total, 1 up, 2 in 2024-04-07T20:48:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:27 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:28.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:28 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3592121965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:28.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:48:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:28.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:28 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3592121965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:28.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:28 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3592121965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:29.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[29851]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:29.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:29.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi177 2024-04-07T20:48:29.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[24852]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi177 2024-04-07T20:48:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:29.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:29 smithi187 ceph-mon[25907]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:29.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:29.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:29 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi177 2024-04-07T20:48:29.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:31 smithi177 ceph-mon[24852]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:31 smithi177 ceph-mon[29851]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:31.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:31 smithi187 ceph-mon[25907]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:33.394 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:33 smithi177 ceph-mon[24852]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:33.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:33 smithi177 ceph-mon[29851]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:33.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:33 smithi187 ceph-mon[25907]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:34.355 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:34 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-07T20:48:34.355 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:34 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:34.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:34 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-07T20:48:34.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:34 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:34 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-07T20:48:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:34 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:35.349 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:35 smithi177 ceph-mon[24852]: Deploying daemon osd.1 on smithi177 2024-04-07T20:48:35.350 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:35 smithi177 ceph-mon[24852]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:35.350 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:35 smithi177 ceph-mon[29851]: Deploying daemon osd.1 on smithi177 2024-04-07T20:48:35.350 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:35 smithi177 ceph-mon[29851]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:35.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:35 smithi187 ceph-mon[25907]: Deploying daemon osd.1 on smithi177 2024-04-07T20:48:35.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:35 smithi187 ceph-mon[25907]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:37.343 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:37 smithi177 ceph-mon[24852]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:37.343 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:37 smithi177 ceph-mon[29851]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:37.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:37 smithi187 ceph-mon[25907]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:39.267 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[29851]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:39.267 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:39.267 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:39.267 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:39.267 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:39.267 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:39.268 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[24852]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:39.268 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:39.268 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:39.268 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:39.268 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:39.268 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:39.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:39 smithi187 ceph-mon[25907]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:39.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:39.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:39.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:39.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:39.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:39.630 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:48:39] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:40 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:40 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:40 smithi177 ceph-mon[24852]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:40.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:40 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:40.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:40 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:40.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:40 smithi177 ceph-mon[29851]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:40.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:40 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:40.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:40 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:40.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:40 smithi187 ceph-mon[25907]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:41.217 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 1 on host 'smithi177' 2024-04-07T20:48:41.715 DEBUG:teuthology.orchestra.run.smithi177:osd.1> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-07T20:48:41.718 INFO:tasks.cephadm:Deploying osd.2 on smithi177 with /dev/vg_nvme/lv_2... 2024-04-07T20:48:41.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi177 to 7520M 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T20:48:42.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[29851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi177 to 7520M 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T20:48:42.403 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:42 smithi177 ceph-mon[24852]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T20:48:42.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:42.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:42.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:42.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:42.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:48:42.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi177 to 7520M 2024-04-07T20:48:42.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:42.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T20:48:42.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:42 smithi187 ceph-mon[25907]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[24852]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[24852]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[24852]: osdmap e11: 2 total, 1 up, 2 in 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[24852]: from='osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[24852]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:43.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[29851]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:43.484 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[29851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-07T20:48:43.484 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[29851]: osdmap e11: 2 total, 1 up, 2 in 2024-04-07T20:48:43.484 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:43.484 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[29851]: from='osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:43.484 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-mon[29851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:43.484 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1[35976]: 2024-04-07T20:48:43.273+0000 7f95ad979700 -1 osd.1 0 waiting for initial osdmap 2024-04-07T20:48:43.484 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:48:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1[35976]: 2024-04-07T20:48:43.276+0000 7f95a8310700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:48:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:43 smithi187 ceph-mon[25907]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:43 smithi187 ceph-mon[25907]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-07T20:48:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:43 smithi187 ceph-mon[25907]: osdmap e11: 2 total, 1 up, 2 in 2024-04-07T20:48:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:43 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:43 smithi187 ceph-mon[25907]: from='osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:43 smithi187 ceph-mon[25907]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:48:43.785 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-07T20:48:43.785 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-07T20:48:43.785 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-07T20:48:43.785 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-07T20:48:43.785 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123182 s, 851 MB/s 2024-04-07T20:48:43.785 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[29851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[29851]: osdmap e12: 2 total, 1 up, 2 in 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[24852]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[24852]: osdmap e12: 2 total, 1 up, 2 in 2024-04-07T20:48:44.386 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:44.387 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:44.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-04-07T20:48:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:44 smithi187 ceph-mon[25907]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:48:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:44 smithi187 ceph-mon[25907]: osdmap e12: 2 total, 1 up, 2 in 2024-04-07T20:48:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587] boot 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: osdmap e13: 2 total, 2 up, 2 in 2024-04-07T20:48:45.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587] boot 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: osdmap e13: 2 total, 2 up, 2 in 2024-04-07T20:48:45.529 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: osd.1 [v2:172.21.15.177:6810/3320710587,v1:172.21.15.177:6811/3320710587] boot 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: osdmap e13: 2 total, 2 up, 2 in 2024-04-07T20:48:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:45 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:48:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:46 smithi187 ceph-mon[25907]: osdmap e14: 2 total, 2 up, 2 in 2024-04-07T20:48:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:46 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:46 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:46 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[24852]: osdmap e14: 2 total, 2 up, 2 in 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[29851]: osdmap e14: 2 total, 2 up, 2 in 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:48:46.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:48:46.605 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:46 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:47.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:47 smithi177 ceph-mon[24852]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:47.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:47 smithi177 ceph-mon[24852]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:47.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:47 smithi177 ceph-mon[29851]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:47.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:47 smithi177 ceph-mon[29851]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:47.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:47 smithi187 ceph-mon[25907]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:47.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:47 smithi187 ceph-mon[25907]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:48:48.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:48:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:49.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2583538368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]: dispatch 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]: dispatch 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]': finished 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: osdmap e15: 3 total, 2 up, 3 in 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3205606127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:49.290 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2583538368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]: dispatch 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]: dispatch 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]': finished 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: osdmap e15: 3 total, 2 up, 3 in 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:48:49.291 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:49 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3205606127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:49.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:49.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:49.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2583538368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]: dispatch 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]: dispatch 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b694bab5-4027-490c-b0cc-fe79ea645bde"}]': finished 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: osdmap e15: 3 total, 2 up, 3 in 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:48:49.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:49 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3205606127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:48:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi177 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: pgmap v61: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:50.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi177 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: pgmap v61: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:50.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi177 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: pgmap v61: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:48:50.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:48:51.545 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:51 smithi177 ceph-mon[29851]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:51.545 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:51 smithi177 ceph-mon[29851]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:48:51.545 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:51 smithi177 ceph-mon[29851]: Cluster is now healthy 2024-04-07T20:48:51.545 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:51 smithi177 ceph-mon[24852]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:51.545 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:51 smithi177 ceph-mon[24852]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:48:51.545 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:51 smithi177 ceph-mon[24852]: Cluster is now healthy 2024-04-07T20:48:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:51 smithi187 ceph-mon[25907]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:51 smithi187 ceph-mon[25907]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:48:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:51 smithi187 ceph-mon[25907]: Cluster is now healthy 2024-04-07T20:48:53.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:53 smithi177 ceph-mon[29851]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:53.282 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:53 smithi177 ceph-mon[24852]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:53.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:53 smithi187 ceph-mon[25907]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:54.628 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:54 smithi177 ceph-mon[29851]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:54.629 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:54 smithi177 ceph-mon[24852]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:54.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:54 smithi187 ceph-mon[25907]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:56.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-07T20:48:56.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:55 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:56.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-07T20:48:56.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:55 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:56.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-07T20:48:56.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:55 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:48:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:56 smithi187 ceph-mon[25907]: Deploying daemon osd.2 on smithi177 2024-04-07T20:48:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:56 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:48:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:56 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:48:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:56 smithi187 ceph-mon[25907]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:57.108 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[29851]: Deploying daemon osd.2 on smithi177 2024-04-07T20:48:57.108 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:48:57.109 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:48:57.109 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[29851]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:57.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[24852]: Deploying daemon osd.2 on smithi177 2024-04-07T20:48:57.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:48:57.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:48:57.109 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:56 smithi177 ceph-mon[24852]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:58.423 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:48:58 smithi177 ceph-mon[24852]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:58.423 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:48:58 smithi177 ceph-mon[29851]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:58.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:48:58 smithi187 ceph-mon[25907]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:48:59.759 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:48:59 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:48:59] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:00.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:00.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:00.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:00 smithi177 ceph-mon[29851]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:00.744 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:00.744 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:00.744 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:00 smithi177 ceph-mon[24852]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:00.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:00.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:00.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:00 smithi187 ceph-mon[25907]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:01.978 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:01.978 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:01.978 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:01.978 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:01.978 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:01 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:02.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:02.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:02.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:02.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:02.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:02.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:02.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:02.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:02.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:02.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:01 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:03.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:03 smithi177 ceph-mon[29851]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:03.407 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:03 smithi177 ceph-mon[24852]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:03.437 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 2 on host 'smithi177' 2024-04-07T20:49:03.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:03 smithi187 ceph-mon[25907]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:04.104 DEBUG:teuthology.orchestra.run.smithi177:osd.2> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-07T20:49:04.107 INFO:tasks.cephadm:Deploying osd.3 on smithi177 with /dev/vg_nvme/lv_1... 2024-04-07T20:49:04.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-07T20:49:04.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi177 to 5013M 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T20:49:04.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[24852]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi177 to 5013M 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T20:49:04.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:04 smithi177 ceph-mon[29851]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:04.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:04.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi177 to 5013M 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T20:49:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:04 smithi187 ceph-mon[25907]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[29851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[29851]: osdmap e16: 3 total, 2 up, 3 in 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[29851]: from='osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[29851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[24852]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[24852]: osdmap e16: 3 total, 2 up, 3 in 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[24852]: from='osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:05.660 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-mon[24852]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:05.661 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2[38909]: 2024-04-07T20:49:05.481+0000 7f0c61334700 -1 osd.2 0 waiting for initial osdmap 2024-04-07T20:49:05.661 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:05 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2[38909]: 2024-04-07T20:49:05.484+0000 7f0c5cccd700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:49:05.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:05 smithi187 ceph-mon[25907]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-07T20:49:05.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:05 smithi187 ceph-mon[25907]: osdmap e16: 3 total, 2 up, 3 in 2024-04-07T20:49:05.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:05 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:05.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:05 smithi187 ceph-mon[25907]: from='osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:05.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:05 smithi187 ceph-mon[25907]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:06.146 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-07T20:49:06.146 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-07T20:49:06.146 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-07T20:49:06.146 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-07T20:49:06.146 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122161 s, 858 MB/s 2024-04-07T20:49:06.146 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-07T20:49:06.723 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[29851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:49:06.723 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[29851]: osdmap e17: 3 total, 2 up, 3 in 2024-04-07T20:49:06.723 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:06.723 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:06.723 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[29851]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:06.724 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[24852]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:49:06.724 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[24852]: osdmap e17: 3 total, 2 up, 3 in 2024-04-07T20:49:06.724 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:06.724 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:06.724 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:06 smithi177 ceph-mon[24852]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:06.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-04-07T20:49:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:06 smithi187 ceph-mon[25907]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:49:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:06 smithi187 ceph-mon[25907]: osdmap e17: 3 total, 2 up, 3 in 2024-04-07T20:49:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:06 smithi187 ceph-mon[25907]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[29851]: osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110] boot 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[29851]: osdmap e18: 3 total, 3 up, 3 in 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:49:07.730 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:07.731 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[24852]: osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110] boot 2024-04-07T20:49:07.731 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[24852]: osdmap e18: 3 total, 3 up, 3 in 2024-04-07T20:49:07.731 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:07 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:07.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:07 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:49:07.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:07 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:49:07.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:07 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:07.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:07 smithi187 ceph-mon[25907]: osd.2 [v2:172.21.15.177:6818/99096110,v1:172.21.15.177:6819/99096110] boot 2024-04-07T20:49:07.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:07 smithi187 ceph-mon[25907]: osdmap e18: 3 total, 3 up, 3 in 2024-04-07T20:49:07.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:07 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:49:08.566 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:08 smithi177 ceph-mon[24852]: osdmap e19: 3 total, 3 up, 3 in 2024-04-07T20:49:08.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:08 smithi177 ceph-mon[29851]: osdmap e19: 3 total, 3 up, 3 in 2024-04-07T20:49:08.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:08 smithi187 ceph-mon[25907]: osdmap e19: 3 total, 3 up, 3 in 2024-04-07T20:49:09.602 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: pgmap v76: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:09.602 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-07T20:49:09.602 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: osdmap e20: 3 total, 3 up, 3 in 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: pgmap v76: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: osdmap e20: 3 total, 3 up, 3 in 2024-04-07T20:49:09.603 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-07T20:49:09.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:09.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:09.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:09.604 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:09.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: pgmap v76: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:09.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-07T20:49:09.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-07T20:49:09.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: osdmap e20: 3 total, 3 up, 3 in 2024-04-07T20:49:09.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-07T20:49:09.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:09.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:09.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:09.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:10.188 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:49:09 smithi177 sudo[40130]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T20:49:10.188 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:49:09 smithi177 sudo[40130]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:10.188 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:49:09 smithi177 sudo[40130]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:10.478 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:49:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:10.478 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40133]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T20:49:10.478 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40133]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:10.478 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40133]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:10.735 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-07T20:49:10.735 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: osdmap e21: 3 total, 3 up, 3 in 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:10.736 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.736 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40166]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T20:49:10.736 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40166]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:10.736 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40166]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: osdmap e21: 3 total, 3 up, 3 in 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:10.737 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:10.990 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40183]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T20:49:10.990 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40183]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:10.990 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40183]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: osdmap e21: 3 total, 3 up, 3 in 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:11.252 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40202]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T20:49:11.253 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:10 smithi177 sudo[40202]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:11.253 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40202]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:11.506 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40228]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T20:49:11.506 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40228]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:11.506 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40221]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T20:49:11.506 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40221]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:11.506 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40221]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:11.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40228]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:12.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:11 smithi177 ceph-mon[29851]: osdmap e22: 3 total, 3 up, 3 in 2024-04-07T20:49:12.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:11 smithi177 ceph-mon[29851]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:49:12.030 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40274]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T20:49:12.030 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40274]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:12.030 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 sudo[40274]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:12.031 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 ceph-mon[24852]: osdmap e22: 3 total, 3 up, 3 in 2024-04-07T20:49:12.031 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:11 smithi177 ceph-mon[24852]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:49:12.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:11 smithi187 ceph-mon[25907]: osdmap e22: 3 total, 3 up, 3 in 2024-04-07T20:49:12.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:11 smithi187 ceph-mon[25907]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:49:12.308 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 sudo[40296]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T20:49:12.308 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 sudo[40296]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:12.308 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 sudo[40296]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:12.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 sudo[40380]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T20:49:12.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 sudo[40380]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:12.650 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 sudo[40380]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:12.802 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 sudo[28295]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T20:49:12.802 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 sudo[28295]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:12.802 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 sudo[28295]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:12.942 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:49:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:12.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi177 2024-04-07T20:49:12.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:12.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi177 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:12.944 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:12.945 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:12 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 sudo[28298]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 sudo[28298]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 sudo[28298]: pam_unix(sudo:session): session closed for user root 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi177 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:49:13.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:12 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[24852]: Cluster is now healthy 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:49:14.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:13 smithi177 ceph-mon[29851]: Cluster is now healthy 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:14.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:14.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:14.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:14.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:49:14.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:13 smithi187 ceph-mon[25907]: Cluster is now healthy 2024-04-07T20:49:15.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3962563912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08cf95d7-da16-4395-ab77-f20b83859262"}]: dispatch 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3962563912' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08cf95d7-da16-4395-ab77-f20b83859262"}]': finished 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[29851]: osdmap e23: 4 total, 3 up, 4 in 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1337584910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3962563912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08cf95d7-da16-4395-ab77-f20b83859262"}]: dispatch 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3962563912' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08cf95d7-da16-4395-ab77-f20b83859262"}]': finished 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[24852]: osdmap e23: 4 total, 3 up, 4 in 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:15.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:14 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1337584910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:49:15.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:14 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3962563912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08cf95d7-da16-4395-ab77-f20b83859262"}]: dispatch 2024-04-07T20:49:15.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:14 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3962563912' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08cf95d7-da16-4395-ab77-f20b83859262"}]': finished 2024-04-07T20:49:15.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:14 smithi187 ceph-mon[25907]: osdmap e23: 4 total, 3 up, 4 in 2024-04-07T20:49:15.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:14 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:15.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:14 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1337584910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:49:16.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:15 smithi177 ceph-mon[24852]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:16.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:15 smithi177 ceph-mon[29851]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:16.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:15 smithi187 ceph-mon[25907]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:18.141 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:18 smithi177 ceph-mon[29851]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 122 KiB/s wr, 8 op/s 2024-04-07T20:49:18.142 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:18 smithi177 ceph-mon[24852]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 122 KiB/s wr, 8 op/s 2024-04-07T20:49:18.350 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:18 smithi187 ceph-mon[25907]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 122 KiB/s wr, 8 op/s 2024-04-07T20:49:20.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:19 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:49:19] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:20.362 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:20 smithi177 ceph-mon[24852]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-04-07T20:49:20.362 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:20.362 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:20 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:20.362 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:20 smithi177 ceph-mon[29851]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-04-07T20:49:20.362 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:20.362 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:20 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:20.387 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:20 smithi187 ceph-mon[25907]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-04-07T20:49:20.387 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:20.387 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:20 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:21.300 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-07T20:49:21.300 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:21.397 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-07T20:49:21.397 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:21.397 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-07T20:49:21.397 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:22.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:22 smithi187 ceph-mon[25907]: Deploying daemon osd.3 on smithi177 2024-04-07T20:49:22.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:22 smithi187 ceph-mon[25907]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 93 KiB/s wr, 6 op/s 2024-04-07T20:49:22.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:22 smithi177 ceph-mon[29851]: Deploying daemon osd.3 on smithi177 2024-04-07T20:49:22.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:22 smithi177 ceph-mon[29851]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 93 KiB/s wr, 6 op/s 2024-04-07T20:49:22.338 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:22 smithi177 ceph-mon[24852]: Deploying daemon osd.3 on smithi177 2024-04-07T20:49:22.338 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:22 smithi177 ceph-mon[24852]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 93 KiB/s wr, 6 op/s 2024-04-07T20:49:24.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:24 smithi177 ceph-mon[24852]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T20:49:24.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:24 smithi177 ceph-mon[29851]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T20:49:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:24 smithi187 ceph-mon[25907]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T20:49:26.235 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[24852]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-04-07T20:49:26.235 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[29851]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:26.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:26 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:26 smithi187 ceph-mon[25907]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-04-07T20:49:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:26 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:26 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:26 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:26 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:26 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:28.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:28 smithi177 ceph-mon[24852]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-07T20:49:28.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:28 smithi177 ceph-mon[29851]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-07T20:49:28.369 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 3 on host 'smithi177' 2024-04-07T20:49:28.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:28 smithi187 ceph-mon[25907]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-07T20:49:29.009 DEBUG:teuthology.orchestra.run.smithi177:osd.3> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-07T20:49:29.012 INFO:tasks.cephadm:Deploying osd.4 on smithi187 with /dev/vg_nvme/lv_4... 2024-04-07T20:49:29.012 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-07T20:49:29.620 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi177 to 3760M 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T20:49:29.621 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[29851]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi177 to 3760M 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T20:49:29.622 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T20:49:29.623 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:29 smithi177 ceph-mon[24852]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:29.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi177 to 3760M 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T20:49:29.675 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:29 smithi187 ceph-mon[25907]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:30.468 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[24852]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[24852]: osdmap e24: 4 total, 3 up, 4 in 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[24852]: from='osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[24852]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[29851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[29851]: osdmap e24: 4 total, 3 up, 4 in 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[29851]: from='osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:30.469 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-mon[29851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:30.469 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:49:30.428+0000 7fa8702b9700 -1 osd.3 0 waiting for initial osdmap 2024-04-07T20:49:30.470 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:49:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:49:30.431+0000 7fa869c4e700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:49:30.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:30 smithi187 ceph-mon[25907]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-07T20:49:30.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:30 smithi187 ceph-mon[25907]: osdmap e24: 4 total, 3 up, 4 in 2024-04-07T20:49:30.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:30 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:30.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:30 smithi187 ceph-mon[25907]: from='osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:30.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:30 smithi187 ceph-mon[25907]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-07T20:49:31.056 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-07T20:49:31.057 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-07T20:49:31.057 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-04-07T20:49:31.057 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-04-07T20:49:31.057 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012971 s, 808 MB/s 2024-04-07T20:49:31.057 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-04-07T20:49:31.641 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:31 smithi187 ceph-mon[25907]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:49:31.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:31 smithi187 ceph-mon[25907]: osdmap e25: 4 total, 3 up, 4 in 2024-04-07T20:49:31.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:31 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:31.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:31 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:31.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:31 smithi187 ceph-mon[25907]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:31.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2024-04-07T20:49:31.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[24852]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:49:31.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[24852]: osdmap e25: 4 total, 3 up, 4 in 2024-04-07T20:49:31.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[24852]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[29851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[29851]: osdmap e25: 4 total, 3 up, 4 in 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:31.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:31 smithi177 ceph-mon[29851]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T20:49:32.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:32 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:49:32.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:32 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:49:32.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:32 smithi187 ceph-mon[25907]: osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573] boot 2024-04-07T20:49:32.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:32 smithi187 ceph-mon[25907]: osdmap e26: 4 total, 4 up, 4 in 2024-04-07T20:49:32.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:32 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:32.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:49:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[24852]: osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573] boot 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[24852]: osdmap e26: 4 total, 4 up, 4 in 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[29851]: osd.3 [v2:172.21.15.177:6826/2388937573,v1:172.21.15.177:6827/2388937573] boot 2024-04-07T20:49:32.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[29851]: osdmap e26: 4 total, 4 up, 4 in 2024-04-07T20:49:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:32 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:49:33.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[24852]: osdmap e27: 4 total, 4 up, 4 in 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi177 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[24852]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[29851]: osdmap e27: 4 total, 4 up, 4 in 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi177 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:33.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:33 smithi177 ceph-mon[29851]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:33.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:33 smithi187 ceph-mon[25907]: osdmap e27: 4 total, 4 up, 4 in 2024-04-07T20:49:33.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:33 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:33.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:33 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi177 2024-04-07T20:49:33.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:33 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:33.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:33 smithi187 ceph-mon[25907]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:34 smithi187 ceph-mon[25907]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:34 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:34 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:34 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:34.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[24852]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:34.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:34.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:34.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:34.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[29851]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:34.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:34.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:34.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:34 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:35.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:35 smithi177 ceph-mon[24852]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:35.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:35 smithi177 ceph-mon[29851]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:35.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:35 smithi187 ceph-mon[25907]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/1518318815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]: dispatch 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]: dispatch 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]': finished 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[24852]: osdmap e28: 5 total, 4 up, 5 in 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/698965272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/1518318815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]: dispatch 2024-04-07T20:49:36.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]: dispatch 2024-04-07T20:49:36.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]': finished 2024-04-07T20:49:36.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[29851]: osdmap e28: 5 total, 4 up, 5 in 2024-04-07T20:49:36.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:36.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:36 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/698965272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:49:36.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:36 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/1518318815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]: dispatch 2024-04-07T20:49:36.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:36 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]: dispatch 2024-04-07T20:49:36.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:36 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "293dbc45-ead8-4fef-bfa0-6dfbd09b5f87"}]': finished 2024-04-07T20:49:36.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:36 smithi187 ceph-mon[25907]: osdmap e28: 5 total, 4 up, 5 in 2024-04-07T20:49:36.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:36 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:36.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:36 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/698965272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:49:38.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:38 smithi187 ceph-mon[25907]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:38.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:38 smithi177 ceph-mon[29851]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:38.407 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:38 smithi177 ceph-mon[24852]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:40.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:49:39] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:40.192 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:40 smithi187 ceph-mon[25907]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:40.192 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:40 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:40.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:40 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:40 smithi177 ceph-mon[24852]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:40 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:40 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:40.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:40 smithi177 ceph-mon[29851]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:40.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:40 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:40.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:40 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:42.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:42 smithi187 ceph-mon[25907]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:42.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:42 smithi177 ceph-mon[24852]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:42.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:42 smithi177 ceph-mon[29851]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:43.407 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:43 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-07T20:49:43.407 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:43 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:43 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-07T20:49:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:43 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:43.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:43 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-07T20:49:43.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:43 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:44.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:44 smithi187 ceph-mon[25907]: Deploying daemon osd.4 on smithi187 2024-04-07T20:49:44.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:44 smithi187 ceph-mon[25907]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:44.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:44 smithi177 ceph-mon[24852]: Deploying daemon osd.4 on smithi187 2024-04-07T20:49:44.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:44 smithi177 ceph-mon[24852]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:44.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:44 smithi177 ceph-mon[29851]: Deploying daemon osd.4 on smithi187 2024-04-07T20:49:44.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:44 smithi177 ceph-mon[29851]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:46.268 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:46 smithi187 ceph-mon[25907]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:46.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:46 smithi177 ceph-mon[24852]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:46.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:46 smithi177 ceph-mon[29851]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:48.274 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:48 smithi187 ceph-mon[25907]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:48.421 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:48 smithi177 ceph-mon[24852]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:48.421 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:48 smithi177 ceph-mon[29851]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:49.536 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:49.537 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:49.537 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:49.537 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:49.537 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:49.537 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:49 smithi187 ceph-mon[25907]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:49.683 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[24852]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-mon[29851]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:50.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:49 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:49:49] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:49:51.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:51.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:50 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:51.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:51.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:50 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:51.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:51.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:50 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:52.052 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:51 smithi187 ceph-mon[25907]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:52.052 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:51 smithi187 ceph-mon[25907]: from='osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T20:49:52.052 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:51 smithi187 ceph-mon[25907]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T20:49:52.086 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 4 on host 'smithi187' 2024-04-07T20:49:52.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:51 smithi177 ceph-mon[29851]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:52.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:51 smithi177 ceph-mon[29851]: from='osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T20:49:52.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:51 smithi177 ceph-mon[29851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T20:49:52.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:51 smithi177 ceph-mon[24852]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:52.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:51 smithi177 ceph-mon[24852]: from='osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T20:49:52.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:51 smithi177 ceph-mon[24852]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T20:49:52.728 DEBUG:teuthology.orchestra.run.smithi187:osd.4> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-07T20:49:52.730 INFO:tasks.cephadm:Deploying osd.5 on smithi187 with /dev/vg_nvme/lv_3... 2024-04-07T20:49:52.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-07T20:49:53.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-07T20:49:53.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: osdmap e29: 5 total, 4 up, 5 in 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi187 to 16064M 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.058 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:49:52.878+0000 7fe23dceb700 -1 osd.4 0 waiting for initial osdmap 2024-04-07T20:49:53.058 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:49:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:49:52.882+0000 7fe238682700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: osdmap e29: 5 total, 4 up, 5 in 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:53.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi187 to 16064M 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: osdmap e29: 5 total, 4 up, 5 in 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi187 to 16064M 2024-04-07T20:49:53.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:53.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:49:54.242 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:53 smithi187 ceph-mon[25907]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:54.242 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:53 smithi187 ceph-mon[25907]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:49:54.242 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:53 smithi187 ceph-mon[25907]: osdmap e30: 5 total, 4 up, 5 in 2024-04-07T20:49:54.242 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:53 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:54.242 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:53 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:54.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[24852]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:54.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[24852]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[24852]: osdmap e30: 5 total, 4 up, 5 in 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[29851]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[29851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[29851]: osdmap e30: 5 total, 4 up, 5 in 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:53 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:54.837 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-07T20:49:54.837 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-07T20:49:54.837 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-04-07T20:49:54.837 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-04-07T20:49:54.837 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012021 s, 872 MB/s 2024-04-07T20:49:54.837 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-04-07T20:49:55.200 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:54 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:49:55.200 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:54 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:49:55.200 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:54 smithi187 ceph-mon[25907]: osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953] boot 2024-04-07T20:49:55.200 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:54 smithi187 ceph-mon[25907]: osdmap e31: 5 total, 5 up, 5 in 2024-04-07T20:49:55.200 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:54 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:55.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:49:55.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[24852]: osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953] boot 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[24852]: osdmap e31: 5 total, 5 up, 5 in 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[29851]: osd.4 [v2:172.21.15.187:6800/2244514953,v1:172.21.15.187:6801/2244514953] boot 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[29851]: osdmap e31: 5 total, 5 up, 5 in 2024-04-07T20:49:55.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:54 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:49:55.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2024-04-07T20:49:56.187 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:55 smithi187 ceph-mon[25907]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:49:56.187 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:55 smithi187 ceph-mon[25907]: osdmap e32: 5 total, 5 up, 5 in 2024-04-07T20:49:56.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:55 smithi177 ceph-mon[24852]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:49:56.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:55 smithi177 ceph-mon[24852]: osdmap e32: 5 total, 5 up, 5 in 2024-04-07T20:49:56.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:55 smithi177 ceph-mon[29851]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:49:56.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:55 smithi177 ceph-mon[29851]: osdmap e32: 5 total, 5 up, 5 in 2024-04-07T20:49:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:56 smithi187 ceph-mon[25907]: osdmap e33: 5 total, 5 up, 5 in 2024-04-07T20:49:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:56 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:49:57.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:56 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:49:57.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:56 smithi177 ceph-mon[29851]: osdmap e33: 5 total, 5 up, 5 in 2024-04-07T20:49:57.160 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:56 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:49:57.160 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:56 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:49:57.160 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:56 smithi177 ceph-mon[24852]: osdmap e33: 5 total, 5 up, 5 in 2024-04-07T20:49:57.160 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:56 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:49:57.160 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:56 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:49:58.142 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:57 smithi187 ceph-mon[25907]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:49:58.143 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:57 smithi187 ceph-mon[25907]: osdmap e34: 5 total, 5 up, 5 in 2024-04-07T20:49:58.143 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:57 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:58.143 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:57 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:58.143 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:57 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[24852]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[24852]: osdmap e34: 5 total, 5 up, 5 in 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[29851]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[29851]: osdmap e34: 5 total, 5 up, 5 in 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:49:58.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:49:58.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:57 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:49:59.160 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:58 smithi187 ceph-mon[25907]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:59.160 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:58 smithi187 ceph-mon[25907]: osdmap e35: 5 total, 5 up, 5 in 2024-04-07T20:49:59.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:58 smithi177 ceph-mon[24852]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:59.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:58 smithi177 ceph-mon[24852]: osdmap e35: 5 total, 5 up, 5 in 2024-04-07T20:49:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:58 smithi177 ceph-mon[29851]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:49:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:58 smithi177 ceph-mon[29851]: osdmap e35: 5 total, 5 up, 5 in 2024-04-07T20:50:00.230 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:49:59] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:00.230 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:00.230 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.230 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/2770074633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]: dispatch 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]: dispatch 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]': finished 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: osdmap e36: 6 total, 5 up, 6 in 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.231 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/2770074633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]: dispatch 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]: dispatch 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]': finished 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: osdmap e36: 6 total, 5 up, 6 in 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:49:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.232 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/2770074633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]: dispatch 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]: dispatch 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a51a7901-eccb-4cc2-abc1-4b3c3d06c627"}]': finished 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: osdmap e36: 6 total, 5 up, 6 in 2024-04-07T20:50:00.233 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:49:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:00.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:50:00] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi187 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: overall HEALTH_OK 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:01.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/4278113584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi187 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: overall HEALTH_OK 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:01.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:01.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:00 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/4278113584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:01.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi187 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: overall HEALTH_OK 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:01.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:00 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/4278113584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:03.360 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:03 smithi187 ceph-mon[25907]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-07T20:50:03.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:03 smithi177 ceph-mon[24852]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-07T20:50:03.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:03 smithi177 ceph-mon[29851]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-07T20:50:05.521 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:05 smithi187 ceph-mon[25907]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-07T20:50:05.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:05 smithi177 ceph-mon[24852]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-07T20:50:05.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:05 smithi177 ceph-mon[29851]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-07T20:50:06.405 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-07T20:50:06.405 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:06.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-07T20:50:06.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:06.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-07T20:50:06.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:07.365 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:07 smithi187 ceph-mon[25907]: Deploying daemon osd.5 on smithi187 2024-04-07T20:50:07.365 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:07 smithi187 ceph-mon[25907]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-07T20:50:07.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:07 smithi177 ceph-mon[24852]: Deploying daemon osd.5 on smithi187 2024-04-07T20:50:07.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:07 smithi177 ceph-mon[24852]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-07T20:50:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:07 smithi177 ceph-mon[29851]: Deploying daemon osd.5 on smithi187 2024-04-07T20:50:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:07 smithi177 ceph-mon[29851]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-07T20:50:09.559 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:09 smithi187 ceph-mon[25907]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-07T20:50:09.682 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:09 smithi177 ceph-mon[24852]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-07T20:50:09.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:09 smithi177 ceph-mon[29851]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-07T20:50:10.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:50:09] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:11.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:11.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:11.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:10 smithi177 ceph-mon[24852]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-07T20:50:11.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:11.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:11.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:10 smithi177 ceph-mon[29851]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-07T20:50:11.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:11.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:11.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:10 smithi187 ceph-mon[25907]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-07T20:50:13.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:13 smithi187 ceph-mon[25907]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-07T20:50:13.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:13.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:13.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:13.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:13.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:13 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:13.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[24852]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[29851]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:13 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:15.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:15 smithi187 ceph-mon[25907]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:15.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:15 smithi177 ceph-mon[24852]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:15.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:15 smithi177 ceph-mon[29851]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:16.168 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 5 on host 'smithi187' 2024-04-07T20:50:16.472 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T20:50:16.472 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T20:50:16.472 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.472 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:16.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:16.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:16.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:16.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:16.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:16 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:16.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:16.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:16 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:16.808 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-07T20:50:16.810 INFO:tasks.cephadm:Deploying osd.6 on smithi187 with /dev/vg_nvme/lv_2... 2024-04-07T20:50:16.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi187 to 8032M 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: osdmap e37: 6 total, 5 up, 6 in 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: from='osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:17.443 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-mon[25907]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:17.444 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:50:17.314+0000 7f91a30a6700 -1 osd.5 0 waiting for initial osdmap 2024-04-07T20:50:17.444 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:50:17 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:50:17.318+0000 7f919d23c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:50:17.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi187 to 8032M 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: osdmap e37: 6 total, 5 up, 6 in 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: from='osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[29851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi187 to 8032M 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: osdmap e37: 6 total, 5 up, 6 in 2024-04-07T20:50:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:17.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: from='osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:17.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:17 smithi177 ceph-mon[24852]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:18.566 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:18 smithi187 ceph-mon[25907]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:50:18.566 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:18 smithi187 ceph-mon[25907]: osdmap e38: 6 total, 5 up, 6 in 2024-04-07T20:50:18.566 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:18 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:18.566 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:18 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:18.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[24852]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:50:18.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[24852]: osdmap e38: 6 total, 5 up, 6 in 2024-04-07T20:50:18.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:18.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[29851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:50:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[29851]: osdmap e38: 6 total, 5 up, 6 in 2024-04-07T20:50:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:18 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012527 s, 837 MB/s 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 2024-04-07T20:50:18.860 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-04-07T20:50:19.486 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:50:19.486 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:50:19.487 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:19.487 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:19.487 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752] boot 2024-04-07T20:50:19.487 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: osdmap e39: 6 total, 6 up, 6 in 2024-04-07T20:50:19.487 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:19 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:19.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2024-04-07T20:50:19.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752] boot 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: osdmap e39: 6 total, 6 up, 6 in 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: osd.5 [v2:172.21.15.187:6808/4121303752,v1:172.21.15.187:6809/4121303752] boot 2024-04-07T20:50:19.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: osdmap e39: 6 total, 6 up, 6 in 2024-04-07T20:50:19.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:19 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:50:20.343 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:20 smithi187 ceph-mon[25907]: osdmap e40: 6 total, 6 up, 6 in 2024-04-07T20:50:20.343 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:20 smithi187 ceph-mon[25907]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:20.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:20 smithi177 ceph-mon[24852]: osdmap e40: 6 total, 6 up, 6 in 2024-04-07T20:50:20.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:20 smithi177 ceph-mon[24852]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:20.759 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:20 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:50:20] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:20.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:20 smithi177 ceph-mon[29851]: osdmap e40: 6 total, 6 up, 6 in 2024-04-07T20:50:20.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:20 smithi177 ceph-mon[29851]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:21.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-07T20:50:21.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: osdmap e41: 6 total, 6 up, 6 in 2024-04-07T20:50:21.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi187 2024-04-07T20:50:21.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:21.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:21.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:21.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:21.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: osdmap e41: 6 total, 6 up, 6 in 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi187 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: osdmap e41: 6 total, 6 up, 6 in 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi187 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:21.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:22 smithi187 ceph-mon[25907]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:50:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:22 smithi187 ceph-mon[25907]: osdmap e42: 6 total, 6 up, 6 in 2024-04-07T20:50:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:22 smithi187 ceph-mon[25907]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:50:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:22 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:50:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:22 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:50:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:22 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:22.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[24852]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[24852]: osdmap e42: 6 total, 6 up, 6 in 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[24852]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[29851]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[29851]: osdmap e42: 6 total, 6 up, 6 in 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[29851]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:50:22.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:22 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: osdmap e43: 6 total, 6 up, 6 in 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:23.608 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:23 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:50:23] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: osdmap e43: 6 total, 6 up, 6 in 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:23.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: osdmap e43: 6 total, 6 up, 6 in 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:23.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:23 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:24.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:24 smithi177 ceph-mon[24852]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:24.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:24 smithi177 ceph-mon[24852]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:50:24.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:24 smithi177 ceph-mon[29851]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:24.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:24 smithi177 ceph-mon[29851]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:50:24.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:24 smithi187 ceph-mon[25907]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:24.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:24 smithi187 ceph-mon[25907]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-07T20:50:25.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/1482652682' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]': finished 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[24852]: osdmap e44: 7 total, 6 up, 7 in 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/4025462436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/1482652682' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]': finished 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[29851]: osdmap e44: 7 total, 6 up, 7 in 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:25.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:25 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/4025462436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:25.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:25 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/1482652682' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]: dispatch 2024-04-07T20:50:25.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:25 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]: dispatch 2024-04-07T20:50:25.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:25 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd893288-9d87-4241-ad53-ad9a8c2b69f6"}]': finished 2024-04-07T20:50:25.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:25 smithi187 ceph-mon[25907]: osdmap e44: 7 total, 6 up, 7 in 2024-04-07T20:50:25.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:25 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:25.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:25 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/4025462436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:26.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:26 smithi177 ceph-mon[24852]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-07T20:50:26.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:26 smithi177 ceph-mon[24852]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-07T20:50:26.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:26 smithi177 ceph-mon[24852]: Cluster is now healthy 2024-04-07T20:50:26.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:26 smithi177 ceph-mon[29851]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-07T20:50:26.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:26 smithi177 ceph-mon[29851]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-07T20:50:26.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:26 smithi177 ceph-mon[29851]: Cluster is now healthy 2024-04-07T20:50:26.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:26 smithi187 ceph-mon[25907]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-07T20:50:26.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:26 smithi187 ceph-mon[25907]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-07T20:50:26.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:26 smithi187 ceph-mon[25907]: Cluster is now healthy 2024-04-07T20:50:29.439 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:29 smithi187 ceph-mon[25907]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:50:29.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:29 smithi177 ceph-mon[29851]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:50:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:29 smithi177 ceph-mon[24852]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:50:30.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:50:30] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:31.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:31 smithi187 ceph-mon[25907]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-07T20:50:31.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:31 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:31.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:31 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:31.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:31 smithi177 ceph-mon[24852]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-07T20:50:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:31 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:31 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:31 smithi177 ceph-mon[29851]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-07T20:50:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:31 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:31 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:32.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:32 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-07T20:50:32.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:32 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:32.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:32 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-07T20:50:32.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:32 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:32.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:32 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-07T20:50:32.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:32 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:33.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:33 smithi177 ceph-mon[24852]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-07T20:50:33.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:33 smithi177 ceph-mon[24852]: Deploying daemon osd.6 on smithi187 2024-04-07T20:50:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:33 smithi177 ceph-mon[29851]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-07T20:50:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:33 smithi177 ceph-mon[29851]: Deploying daemon osd.6 on smithi187 2024-04-07T20:50:33.543 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:33 smithi187 ceph-mon[25907]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-07T20:50:33.543 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:33 smithi187 ceph-mon[25907]: Deploying daemon osd.6 on smithi187 2024-04-07T20:50:35.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:35 smithi187 ceph-mon[25907]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T20:50:35.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:35 smithi177 ceph-mon[24852]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T20:50:35.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:35 smithi177 ceph-mon[29851]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T20:50:37.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:37 smithi177 ceph-mon[24852]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:37.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:37 smithi177 ceph-mon[29851]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:37.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:37 smithi187 ceph-mon[25907]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:39.129 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:39 smithi187 ceph-mon[25907]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:39.129 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:39.129 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:39.129 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:39.129 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:39.129 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:39 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:39.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[24852]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[29851]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:39 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:41.281 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 6 on host 'smithi187' 2024-04-07T20:50:41.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:41 smithi177 ceph-mon[24852]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:41.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:41 smithi177 ceph-mon[29851]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:41.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:41 smithi187 ceph-mon[25907]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:41.940 DEBUG:teuthology.orchestra.run.smithi187:osd.6> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.6.service 2024-04-07T20:50:41.943 INFO:tasks.cephadm:Deploying osd.7 on smithi187 with /dev/vg_nvme/lv_1... 2024-04-07T20:50:41.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-07T20:50:42.529 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: Adjusting osd_memory_target on smithi187 to 5354M 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T20:50:42.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:42 smithi187 ceph-mon[25907]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T20:50:42.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: Adjusting osd_memory_target on smithi187 to 5354M 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T20:50:42.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[24852]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: Adjusting osd_memory_target on smithi187 to 5354M 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T20:50:42.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:42 smithi177 ceph-mon[29851]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T20:50:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-mon[25907]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-mon[25907]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-07T20:50:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-mon[25907]: osdmap e45: 7 total, 6 up, 7 in 2024-04-07T20:50:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-mon[25907]: from='osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-mon[25907]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:43.557 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:50:43.318+0000 7f417c89b700 -1 osd.6 0 waiting for initial osdmap 2024-04-07T20:50:43.557 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:50:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:50:43.325+0000 7f4176a31700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T20:50:43.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:50:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[24852]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[24852]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[24852]: osdmap e45: 7 total, 6 up, 7 in 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[24852]: from='osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[24852]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[29851]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[29851]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[29851]: osdmap e45: 7 total, 6 up, 7 in 2024-04-07T20:50:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:43.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[29851]: from='osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:43.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:43 smithi177 ceph-mon[29851]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:50:44.046 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-07T20:50:44.046 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-07T20:50:44.047 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-04-07T20:50:44.047 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-04-07T20:50:44.047 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128834 s, 814 MB/s 2024-04-07T20:50:44.047 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-04-07T20:50:44.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:50:44.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: osdmap e46: 7 total, 6 up, 7 in 2024-04-07T20:50:44.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:44.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:44.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.683 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.683 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:44.683 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:44.683 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:44.683 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:44 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi187:vg_nvme/lv_1 2024-04-07T20:50:44.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: osdmap e46: 7 total, 6 up, 7 in 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: osdmap e46: 7 total, 6 up, 7 in 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:44.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:44.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:44.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:44.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:44.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:44.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:44 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi187 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152] boot 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: osdmap e47: 7 total, 7 up, 7 in 2024-04-07T20:50:45.628 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:45 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:45.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:50:45.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi187 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152] boot 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: osdmap e47: 7 total, 7 up, 7 in 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi187 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: osd.6 [v2:172.21.15.187:6816/3582202152,v1:172.21.15.187:6817/3582202152] boot 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: osdmap e47: 7 total, 7 up, 7 in 2024-04-07T20:50:45.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:45 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:50:46.625 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:46 smithi187 ceph-mon[25907]: osdmap e48: 7 total, 7 up, 7 in 2024-04-07T20:50:46.625 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:46 smithi187 ceph-mon[25907]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:46.625 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:46 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:46.626 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:46 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:46.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:50:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[24852]: osdmap e48: 7 total, 7 up, 7 in 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[24852]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[29851]: osdmap e48: 7 total, 7 up, 7 in 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[29851]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:46.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:46 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:47.651 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:47 smithi187 ceph-mon[25907]: osdmap e49: 7 total, 7 up, 7 in 2024-04-07T20:50:47.651 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:47 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:47.651 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:47 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:47.651 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:47 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:47.651 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:47 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:47.651 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:47 smithi187 ceph-mon[25907]: osdmap e50: 7 total, 7 up, 7 in 2024-04-07T20:50:47.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[24852]: osdmap e49: 7 total, 7 up, 7 in 2024-04-07T20:50:47.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[24852]: osdmap e50: 7 total, 7 up, 7 in 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[29851]: osdmap e49: 7 total, 7 up, 7 in 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:47.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:47 smithi177 ceph-mon[29851]: osdmap e50: 7 total, 7 up, 7 in 2024-04-07T20:50:49.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:49 smithi187 ceph-mon[25907]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:49.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:49 smithi187 ceph-mon[25907]: osdmap e51: 7 total, 7 up, 7 in 2024-04-07T20:50:49.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:50:49.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:50:49.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:49 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[24852]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[24852]: osdmap e51: 7 total, 7 up, 7 in 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[29851]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[29851]: osdmap e51: 7 total, 7 up, 7 in 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T20:50:49.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:49 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:50.208 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:50 smithi177 ceph-mon[24852]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:50:50.208 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:50 smithi177 ceph-mon[29851]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:50:50.232 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:50 smithi187 ceph-mon[25907]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:50:50.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:50:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:50:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:50:51.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:51 smithi177 ceph-mon[24852]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:51.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:51 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:51.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:51 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:51.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:51 smithi177 ceph-mon[29851]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:51.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:51 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:51.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:51 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:51.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:51 smithi187 ceph-mon[25907]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:50:51.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:51 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:51.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:51 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:50:52.478 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:52 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/1276693883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]: dispatch 2024-04-07T20:50:52.478 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:52 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]: dispatch 2024-04-07T20:50:52.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:52 smithi187 ceph-mon[25907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]': finished 2024-04-07T20:50:52.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:52 smithi187 ceph-mon[25907]: osdmap e52: 8 total, 7 up, 8 in 2024-04-07T20:50:52.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:52 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:50:52.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/1276693883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]: dispatch 2024-04-07T20:50:52.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]: dispatch 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[24852]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]': finished 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[24852]: osdmap e52: 8 total, 7 up, 8 in 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/1276693883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]: dispatch 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]: dispatch 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[29851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70a167aa-8959-4adf-864a-524538d14f75"}]': finished 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[29851]: osdmap e52: 8 total, 7 up, 8 in 2024-04-07T20:50:52.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:52 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:50:53.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:53 smithi177 ceph-mon[24852]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:50:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:53 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/1105043332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:53 smithi177 ceph-mon[29851]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:50:53.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:53 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/1105043332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:53.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:53 smithi187 ceph-mon[25907]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:50:53.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:53 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/1105043332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T20:50:55.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:55 smithi187 ceph-mon[25907]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-07T20:50:55.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:55 smithi177 ceph-mon[29851]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-07T20:50:55.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:55 smithi177 ceph-mon[24852]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-07T20:50:57.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:57 smithi177 ceph-mon[24852]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T20:50:57.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:50:57.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:57 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:50:57.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:57 smithi177 ceph-mon[29851]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T20:50:57.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:57 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:50:57.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:57 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:50:57.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:57 smithi187 ceph-mon[25907]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T20:50:57.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:57 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:50:57.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:57 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:50:59.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:59 smithi177 ceph-mon[24852]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T20:50:59.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-07T20:50:59.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:50:59 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:59.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:59 smithi177 ceph-mon[29851]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T20:50:59.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-07T20:50:59.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:50:59 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:50:59.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:59 smithi187 ceph-mon[25907]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T20:50:59.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-07T20:50:59.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:50:59 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:00.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:00 smithi177 ceph-mon[29851]: Deploying daemon osd.7 on smithi187 2024-04-07T20:51:00.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:00 smithi177 ceph-mon[24852]: Deploying daemon osd.7 on smithi187 2024-04-07T20:51:00.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:00 smithi187 ceph-mon[25907]: Deploying daemon osd.7 on smithi187 2024-04-07T20:51:01.494 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:01 smithi187 ceph-mon[25907]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T20:51:01.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:01 smithi177 ceph-mon[24852]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T20:51:01.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:01 smithi177 ceph-mon[29851]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T20:51:03.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:03 smithi187 ceph-mon[25907]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:03.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:03 smithi177 ceph-mon[29851]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:03.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:03 smithi177 ceph-mon[24852]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:04 smithi187 ceph-mon[25907]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:04 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:05.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[24852]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[29851]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:05.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:05.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:04 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:06.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:51:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:07.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:06 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:07.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:06 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:08.030 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:07 smithi187 ceph-mon[25907]: from='osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T20:51:08.030 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:07 smithi187 ceph-mon[25907]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T20:51:08.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:07 smithi177 ceph-mon[24852]: from='osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T20:51:08.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:07 smithi177 ceph-mon[24852]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T20:51:08.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:07 smithi177 ceph-mon[29851]: from='osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T20:51:08.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:07 smithi177 ceph-mon[29851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T20:51:08.366 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 7 on host 'smithi187' 2024-04-07T20:51:08.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:51:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:09.037 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: osdmap e53: 8 total, 7 up, 8 in 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.038 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: osdmap e53: 8 total, 7 up, 8 in 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:09.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:09.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:08 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: osdmap e53: 8 total, 7 up, 8 in 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:09.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:09.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:09.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:09.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:08 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:09.545 DEBUG:teuthology.orchestra.run.smithi187:osd.7> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.7.service 2024-04-07T20:51:09.548 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-07T20:51:09.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: osdmap e54: 8 total, 7 up, 8 in 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180] boot 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: osdmap e55: 8 total, 8 up, 8 in 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: osdmap e54: 8 total, 7 up, 8 in 2024-04-07T20:51:10.007 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180] boot 2024-04-07T20:51:10.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: osdmap e55: 8 total, 8 up, 8 in 2024-04-07T20:51:10.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:09 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-04-07T20:51:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: osdmap e54: 8 total, 7 up, 8 in 2024-04-07T20:51:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: osd.7 [v2:172.21.15.187:6824/1477627180,v1:172.21.15.187:6825/1477627180] boot 2024-04-07T20:51:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: osdmap e55: 8 total, 8 up, 8 in 2024-04-07T20:51:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:09 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:10.595 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:51:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[29851]: purged_snaps scrub starts 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[29851]: purged_snaps scrub ok 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[29851]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[29851]: osdmap e56: 8 total, 8 up, 8 in 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[24852]: purged_snaps scrub starts 2024-04-07T20:51:10.846 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[24852]: purged_snaps scrub ok 2024-04-07T20:51:10.847 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[24852]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:10.847 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:10.847 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:10.847 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:10 smithi177 ceph-mon[24852]: osdmap e56: 8 total, 8 up, 8 in 2024-04-07T20:51:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:10 smithi187 ceph-mon[25907]: purged_snaps scrub starts 2024-04-07T20:51:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:10 smithi187 ceph-mon[25907]: purged_snaps scrub ok 2024-04-07T20:51:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:10 smithi187 ceph-mon[25907]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T20:51:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:10 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:10 smithi187 ceph-mon[25907]: osdmap e56: 8 total, 8 up, 8 in 2024-04-07T20:51:12.733 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:51:13.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:12 smithi177 ceph-mon[24852]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:13.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:12 smithi177 ceph-mon[24852]: osdmap e57: 8 total, 8 up, 8 in 2024-04-07T20:51:13.008 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:12 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/25761066' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T20:51:13.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:12 smithi177 ceph-mon[29851]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:13.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:12 smithi177 ceph-mon[29851]: osdmap e57: 8 total, 8 up, 8 in 2024-04-07T20:51:13.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:12 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/25761066' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T20:51:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:12 smithi187 ceph-mon[25907]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:12 smithi187 ceph-mon[25907]: osdmap e57: 8 total, 8 up, 8 in 2024-04-07T20:51:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:12 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/25761066' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T20:51:13.418 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1712523069,"num_in_osds":8,"osd_in_since":1712523051,"num_remapped_pgs":1} 2024-04-07T20:51:13.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-07T20:51:13.633 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:51:14.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:14 smithi177 ceph-mon[29851]: osdmap e58: 8 total, 8 up, 8 in 2024-04-07T20:51:14.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:14 smithi177 ceph-mon[24852]: osdmap e58: 8 total, 8 up, 8 in 2024-04-07T20:51:14.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:14 smithi187 ceph-mon[25907]: osdmap e58: 8 total, 8 up, 8 in 2024-04-07T20:51:15.363 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:15 smithi177 ceph-mon[29851]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:15.363 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:15 smithi177 ceph-mon[29851]: osdmap e59: 8 total, 8 up, 8 in 2024-04-07T20:51:15.363 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:15 smithi177 ceph-mon[24852]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:15.363 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:15 smithi177 ceph-mon[24852]: osdmap e59: 8 total, 8 up, 8 in 2024-04-07T20:51:15.548 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:51:15.549 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":59,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","created":"2024-04-07T20:46:02.894482+0000","modified":"2024-04-07T20:51:14.104092+0000","last_up_change":"2024-04-07T20:51:09.781055+0000","last_in_change":"2024-04-07T20:50:51.817693+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-07T20:49:08.521467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"108f20e3-d403-4cb5-8ef2-e62d0f73fb65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6803","nonce":3585323055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6805","nonce":3585323055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6809","nonce":3585323055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6807","nonce":3585323055}]},"public_addr":"172.21.15.177:6803/3585323055","cluster_addr":"172.21.15.177:6805/3585323055","heartbeat_back_addr":"172.21.15.177:6809/3585323055","heartbeat_front_addr":"172.21.15.177:6807/3585323055","state":["exists","up"]},{"osd":1,"uuid":"30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6811","nonce":3320710587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6813","nonce":3320710587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6817","nonce":3320710587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6815","nonce":3320710587}]},"public_addr":"172.21.15.177:6811/3320710587","cluster_addr":"172.21.15.177:6813/3320710587","heartbeat_back_addr":"172.21.15.177:6817/3320710587","heartbeat_front_addr":"172.21.15.177:6815/3320710587","state":["exists","up"]},{"osd":2,"uuid":"b694bab5-4027-490c-b0cc-fe79ea645bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6819","nonce":99096110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6821","nonce":99096110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6825","nonce":99096110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6823","nonce":99096110}]},"public_addr":"172.21.15.177:6819/99096110","cluster_addr":"172.21.15.177:6821/99096110","heartbeat_back_addr":"172.21.15.177:6825/99096110","heartbeat_front_addr":"172.21.15.177:6823/99096110","state":["exists","up"]},{"osd":3,"uuid":"08cf95d7-da16-4395-ab77-f20b83859262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6827","nonce":2388937573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6829","nonce":2388937573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6833","nonce":2388937573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6831","nonce":2388937573}]},"public_addr":"172.21.15.177:6827/2388937573","cluster_addr":"172.21.15.177:6829/2388937573","heartbeat_back_addr":"172.21.15.177:6833/2388937573","heartbeat_front_addr":"172.21.15.177:6831/2388937573","state":["exists","up"]},{"osd":4,"uuid":"293dbc45-ead8-4fef-bfa0-6dfbd09b5f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6801","nonce":2244514953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6803","nonce":2244514953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6807","nonce":2244514953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6805","nonce":2244514953}]},"public_addr":"172.21.15.187:6801/2244514953","cluster_addr":"172.21.15.187:6803/2244514953","heartbeat_back_addr":"172.21.15.187:6807/2244514953","heartbeat_front_addr":"172.21.15.187:6805/2244514953","state":["exists","up"]},{"osd":5,"uuid":"a51a7901-eccb-4cc2-abc1-4b3c3d06c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6809","nonce":4121303752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6811","nonce":4121303752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6815","nonce":4121303752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6813","nonce":4121303752}]},"public_addr":"172.21.15.187:6809/4121303752","cluster_addr":"172.21.15.187:6811/4121303752","heartbeat_back_addr":"172.21.15.187:6815/4121303752","heartbeat_front_addr":"172.21.15.187:6813/4121303752","state":["exists","up"]},{"osd":6,"uuid":"bd893288-9d87-4241-ad53-ad9a8c2b69f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6817","nonce":3582202152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6819","nonce":3582202152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6823","nonce":3582202152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6821","nonce":3582202152}]},"public_addr":"172.21.15.187:6817/3582202152","cluster_addr":"172.21.15.187:6819/3582202152","heartbeat_back_addr":"172.21.15.187:6823/3582202152","heartbeat_front_addr":"172.21.15.187:6821/3582202152","state":["exists","up"]},{"osd":7,"uuid":"70a167aa-8959-4adf-864a-524538d14f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6825","nonce":1477627180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6827","nonce":1477627180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6831","nonce":1477627180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6829","nonce":1477627180}]},"public_addr":"172.21.15.187:6825/1477627180","cluster_addr":"172.21.15.187:6827/1477627180","heartbeat_back_addr":"172.21.15.187:6831/1477627180","heartbeat_front_addr":"172.21.15.187:6829/1477627180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:48:21.191935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:48:42.771552+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:04.846712+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:29.672325+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:52.446119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:50:16.546464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:50:42.572341+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:51:07.874882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:0/3514106323":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/851605445":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/1872289908":"2024-04-08T20:46:56.140824+0000","172.21.15.177:6800/3006842145":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/1166756411":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6801/3006842145":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/646064095":"2024-04-08T20:46:35.136972+0000","172.21.15.177:0/3203112426":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6800/3045474916":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6801/3045474916":"2024-04-08T20:46:35.136972+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-07T20:51:15.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:15 smithi187 ceph-mon[25907]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:15.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:15 smithi187 ceph-mon[25907]: osdmap e59: 8 total, 8 up, 8 in 2024-04-07T20:51:16.257 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-07T20:49:08.521467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-07T20:51:16.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-07T20:51:16.468 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:51:16.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:16 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/4004433818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:16.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:16 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4004433818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:16.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:16 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/4004433818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:17.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:17 smithi177 ceph-mon[24852]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-07T20:51:17.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:17 smithi177 ceph-mon[29851]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-07T20:51:17.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:17 smithi187 ceph-mon[25907]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-07T20:51:18.321 INFO:teuthology.orchestra.run.smithi177.stdout:pg_num: 1 2024-04-07T20:51:19.010 INFO:tasks.cephadm:Adding prometheus.a on smithi187 2024-04-07T20:51:19.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi187=a' 2024-04-07T20:51:19.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:19 smithi177 ceph-mon[24852]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:51:19.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:19 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3841771989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-07T20:51:19.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:19 smithi177 ceph-mon[29851]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:51:19.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:19 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3841771989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-07T20:51:19.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:19 smithi187 ceph-mon[25907]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-07T20:51:19.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:19 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3841771989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-07T20:51:21.063 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled prometheus update... 2024-04-07T20:51:21.472 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-07T20:51:21.472 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:21.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:21.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:21.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:21.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:21.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:21 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:21.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' 2024-04-07T20:51:21.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:21 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:21.802 DEBUG:teuthology.orchestra.run.smithi187:prometheus.a> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@prometheus.a.service 2024-04-07T20:51:21.805 INFO:tasks.cephadm:Adding node-exporter.a on smithi177 2024-04-07T20:51:21.805 INFO:tasks.cephadm:Adding node-exporter.b on smithi187 2024-04-07T20:51:21.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi177=a;smithi187=b' 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-mon[25907]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-mon[25907]: Saving service prometheus spec with placement smithi187=a;count:1 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-mon[25907]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-mon[25907]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ignoring --setuser ceph since I am not root 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ignoring --setgroup ceph since I am not root 2024-04-07T20:51:22.424 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-mgr[27327]: -- 172.21.15.187:0/4172579122 <== mon.2 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560bac1dc340 con 0x560bac1b5000 2024-04-07T20:51:22.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[29851]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:22.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[29851]: Saving service prometheus spec with placement smithi187=a;count:1 2024-04-07T20:51:22.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[29851]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[29851]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[24852]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[24852]: Saving service prometheus spec with placement smithi187=a;count:1 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[24852]: from='mgr.14152 172.21.15.177:0/1949758734' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mon[24852]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setuser ceph since I am not root 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setgroup ceph since I am not root 2024-04-07T20:51:22.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-mgr[25089]: -- 172.21.15.177:0/2629301978 <== mon.1 v2:172.21.15.177:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d2e3310340 con 0x55d2e32e8c00 2024-04-07T20:51:22.688 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:22.600+0000 7f0ff71bf000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:51:23.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:22.590+0000 7f8a9df75000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:51:23.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:22.678+0000 7f8a9df75000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:51:23.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:22.689+0000 7f0ff71bf000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:51:23.448 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.287+0000 7f0ff71bf000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:51:23.448 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.377+0000 7f0ff71bf000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:51:23.577 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.259+0000 7f8a9df75000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:51:23.577 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.347+0000 7f8a9df75000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:51:23.577 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.429+0000 7f8a9df75000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:51:23.577 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.577+0000 7f8a9df75000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:51:23.699 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.464+0000 7f0ff71bf000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:51:23.700 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.617+0000 7f0ff71bf000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:51:23.700 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.700+0000 7f0ff71bf000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:51:23.848 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.658+0000 7f8a9df75000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:51:23.849 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.767+0000 7f8a9df75000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:51:24.052 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.808+0000 7f0ff71bf000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:51:24.052 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:23.891+0000 7f0ff71bf000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:51:24.052 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:24 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:24.053+0000 7f0ff71bf000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:51:24.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:23.849+0000 7f8a9df75000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:51:24.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:24 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:24.011+0000 7f8a9df75000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:51:24.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:24 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:24.096+0000 7f8a9df75000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:51:24.306 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:24 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:24.138+0000 7f0ff71bf000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:51:25.428 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:25 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:25.109+0000 7f8a9df75000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:51:25.428 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:25 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:25.239+0000 7f8a9df75000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:51:25.428 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:25 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:25.428+0000 7f8a9df75000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:51:25.498 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:25.189+0000 7f0ff71bf000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:51:25.499 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:25.315+0000 7f0ff71bf000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:51:25.499 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:25.499+0000 7f0ff71bf000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:51:25.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:25 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:25.516+0000 7f8a9df75000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:51:25.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:25 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:25.650+0000 7f8a9df75000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:51:25.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:25.585+0000 7f0ff71bf000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:51:25.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:25.719+0000 7f0ff71bf000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:51:26.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:25 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:25.969+0000 7f8a9df75000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:51:26.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:26 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:26.190+0000 7f8a9df75000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:51:26.306 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:26.037+0000 7f0ff71bf000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:51:26.306 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:26.253+0000 7f0ff71bf000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:51:26.699 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:26.336+0000 7f0ff71bf000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:51:26.699 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:26.513+0000 7f0ff71bf000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:51:26.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:26 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:26.275+0000 7f8a9df75000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:51:26.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:26 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:26.444+0000 7f8a9df75000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:51:27.654 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:27 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:27.359+0000 7f8a9df75000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:51:27.654 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:27 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:27.654+0000 7f8a9df75000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:51:27.707 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:27.429+0000 7f0ff71bf000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:51:27.913 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:27 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:27.754+0000 7f8a9df75000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:51:27.913 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:27 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:27.912+0000 7f8a9df75000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:51:27.980 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:27.730+0000 7f0ff71bf000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:51:27.980 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:27.825+0000 7f0ff71bf000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:51:27.980 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:27.980+0000 7f0ff71bf000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:51:28.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: Active manager daemon y restarted 2024-04-07T20:51:28.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: Activating manager daemon y 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: osdmap e60: 8 total, 8 up, 8 in 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: Standby manager daemon x restarted 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: Standby manager daemon x started 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: Active manager daemon y restarted 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: Activating manager daemon y 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: osdmap e60: 8 total, 8 up, 8 in 2024-04-07T20:51:28.195 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: Standby manager daemon x restarted 2024-04-07T20:51:28.196 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: Standby manager daemon x started 2024-04-07T20:51:28.196 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-07T20:51:28.196 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:51:28.196 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-07T20:51:28.196 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:51:28.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: Active manager daemon y restarted 2024-04-07T20:51:28.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: Activating manager daemon y 2024-04-07T20:51:28.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: osdmap e60: 8 total, 8 up, 8 in 2024-04-07T20:51:28.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: Standby manager daemon x restarted 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: Standby manager daemon x started 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.187:0/1546458895' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: [07/Apr/2024:20:51:27] ENGINE Bus STARTING 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: CherryPy Checker: 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: The Application mounted at '' has an empty config. 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: [07/Apr/2024:20:51:28] ENGINE Serving on http://:::9283 2024-04-07T20:51:28.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: [07/Apr/2024:20:51:28] ENGINE Bus STARTED 2024-04-07T20:51:28.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:51:28] ENGINE Bus STARTING 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: CherryPy Checker: 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: The Application mounted at '' has an empty config. 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:51:28] ENGINE Serving on http://:::9283 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:51:28] ENGINE Bus STARTED 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:51:28] ENGINE Bus STARTING 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:51:28] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:51:28.777 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:51:28] ENGINE Bus STARTED 2024-04-07T20:51:29.129 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled node-exporter update... 2024-04-07T20:51:29.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: mgrmap e17: y(active, starting, since 0.181296s), standbys: x 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:29.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: Manager daemon y is now available 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]': finished 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]': finished 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:51:29.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: mgrmap e17: y(active, starting, since 0.181296s), standbys: x 2024-04-07T20:51:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:51:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:51:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:51:29.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: Manager daemon y is now available 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: mgrmap e17: y(active, starting, since 0.181296s), standbys: x 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:51:29.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: Manager daemon y is now available 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]': finished 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]': finished 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]': finished 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-04-07T20:51:29.514 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]': finished 2024-04-07T20:51:29.515 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:29.515 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:51:29.515 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:51:29.515 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:51:29.515 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:51:29.515 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.052 DEBUG:teuthology.orchestra.run.smithi177:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.a.service 2024-04-07T20:51:30.056 DEBUG:teuthology.orchestra.run.smithi187:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.b.service 2024-04-07T20:51:30.058 INFO:tasks.cephadm:Adding alertmanager.a on smithi177 2024-04-07T20:51:30.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi177=a' 2024-04-07T20:51:30.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-07T20:51:30.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: Updating smithi187:/etc/ceph/ceph.conf 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: [07/Apr/2024:20:51:28] ENGINE Bus STARTING 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: [07/Apr/2024:20:51:28] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: [07/Apr/2024:20:51:28] ENGINE Bus STARTED 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: mgrmap e18: y(active, since 1.18474s), standbys: x 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi177=a;smithi187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: Saving service node-exporter spec with placement smithi177=a;smithi187=b;count:2 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:30.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: Updating smithi187:/etc/ceph/ceph.conf 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: [07/Apr/2024:20:51:28] ENGINE Bus STARTING 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: [07/Apr/2024:20:51:28] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: [07/Apr/2024:20:51:28] ENGINE Bus STARTED 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:51:30.435 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: mgrmap e18: y(active, since 1.18474s), standbys: x 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi177=a;smithi187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: Saving service node-exporter spec with placement smithi177=a;smithi187=b;count:2 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: Updating smithi187:/etc/ceph/ceph.conf 2024-04-07T20:51:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: [07/Apr/2024:20:51:28] ENGINE Bus STARTING 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: [07/Apr/2024:20:51:28] ENGINE Serving on https://172.21.15.177:7150 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: [07/Apr/2024:20:51:28] ENGINE Bus STARTED 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: mgrmap e18: y(active, since 1.18474s), standbys: x 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi177=a;smithi187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: Saving service node-exporter spec with placement smithi177=a;smithi187=b;count:2 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:30.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T20:51:30.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:30 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:51:30] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:31.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:31 smithi187 ceph-mon[25907]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:31.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:31 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T20:51:31.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:31 smithi187 ceph-mon[25907]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-07T20:51:31.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:31 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:31.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:31 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:31.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[29851]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[29851]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[24852]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[24852]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:31 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:32.157 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled alertmanager update... 2024-04-07T20:51:32.471 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:32 smithi177 ceph-mon[29851]: Deploying daemon prometheus.a on smithi187 2024-04-07T20:51:32.471 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:32 smithi177 ceph-mon[24852]: Deploying daemon prometheus.a on smithi187 2024-04-07T20:51:32.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:32 smithi187 ceph-mon[25907]: Deploying daemon prometheus.a on smithi187 2024-04-07T20:51:32.978 DEBUG:teuthology.orchestra.run.smithi177:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@alertmanager.a.service 2024-04-07T20:51:32.981 INFO:tasks.cephadm:Adding grafana.a on smithi187 2024-04-07T20:51:32.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph orch apply grafana '1;smithi187=a' 2024-04-07T20:51:33.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:33 smithi177 ceph-mon[29851]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:33.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:33 smithi177 ceph-mon[24852]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:33.286 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:33 smithi187 ceph-mon[25907]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:33.286 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:34.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:34 smithi177 ceph-mon[29851]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:34.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:34 smithi177 ceph-mon[29851]: Saving service alertmanager spec with placement smithi177=a;count:1 2024-04-07T20:51:34.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:34 smithi177 ceph-mon[24852]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:34.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:34 smithi177 ceph-mon[24852]: Saving service alertmanager spec with placement smithi177=a;count:1 2024-04-07T20:51:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:34 smithi187 ceph-mon[25907]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:34 smithi187 ceph-mon[25907]: Saving service alertmanager spec with placement smithi177=a;count:1 2024-04-07T20:51:35.405 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:35 smithi187 ceph-mon[25907]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:35.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:35 smithi177 ceph-mon[29851]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:35.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:35 smithi177 ceph-mon[24852]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:35.707 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled grafana update... 2024-04-07T20:51:36.821 DEBUG:teuthology.orchestra.run.smithi187:grafana.a> sudo journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@grafana.a.service 2024-04-07T20:51:36.823 INFO:tasks.cephadm:Setting up client nodes... 2024-04-07T20:51:36.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[24852]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[24852]: Saving service grafana spec with placement smithi187=a;count:1 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[24852]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[29851]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[29851]: Saving service grafana spec with placement smithi187=a;count:1 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:37.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:36 smithi177 ceph-mon[29851]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:37.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:36 smithi187 ceph-mon[25907]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:51:37.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:36 smithi187 ceph-mon[25907]: Saving service grafana spec with placement smithi187=a;count:1 2024-04-07T20:51:37.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:36 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:37.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:36 smithi187 ceph-mon[25907]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:38.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:38 smithi177 ceph-mon[29851]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:38.395 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:38 smithi177 ceph-mon[24852]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:38.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:38 smithi187 ceph-mon[25907]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:38.861 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-04-07T20:51:38.861 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBaBxNm580yMxAAsJBRMN0EU6rtVxPbWpAo+w== 2024-04-07T20:51:39.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:39 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3047702642' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:39 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:39 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T20:51:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:39 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3047702642' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:39 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:39.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:39 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T20:51:39.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:39 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3047702642' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:39.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:39 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:39.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:39 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T20:51:39.566 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-07T20:51:39.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-07T20:51:39.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-07T20:51:39.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-07T20:51:40.296 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:40 smithi187 ceph-mon[25907]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:40.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:40 smithi177 ceph-mon[24852]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:40.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:40 smithi177 ceph-mon[29851]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:43.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:43 smithi177 ceph-mon[24852]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:43.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:43 smithi177 ceph-mon[29851]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:43.456 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:43 smithi187 ceph-mon[25907]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:44.488 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:44 smithi187 ceph-mon[25907]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:44.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:44 smithi177 ceph-mon[24852]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:44.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:44 smithi177 ceph-mon[29851]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:44.806 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:44 smithi187 systemd[1]: Starting Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:51:44.832 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2024-04-07T20:51:44.832 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQBgBxNmCxk+MRAAoIqwnRuHWdAdD8ESyalcLw== 2024-04-07T20:51:45.098 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:44 smithi187 podman[42077]: 2024-04-07 20:51:44.846937353 +0000 UTC m=+0.022096979 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-07T20:51:45.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:44 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:51:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:45.359 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 podman[42077]: 2024-04-07 20:51:45.233532641 +0000 UTC m=+0.408692258 container create e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:51:45.802 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 podman[42077]: 2024-04-07 20:51:45.460728178 +0000 UTC m=+0.635887796 container init e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:51:45.802 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 podman[42077]: 2024-04-07 20:51:45.463345555 +0000 UTC m=+0.638505173 container start e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:51:45.802 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.487Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-07T20:51:45.802 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.487Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.487Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.487Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi187 (none))" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.487Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.487Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.489Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.489Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.490Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.492Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.492Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.669µs 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.492Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.492Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.492Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.621µs wal_replay_duration=131.875µs total_replay_duration=166.173µs 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.493Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-07T20:51:45.803 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.493Z caller=main.go:947 level=info msg="TSDB started" 2024-04-07T20:51:45.804 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.493Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-07T20:51:45.804 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.507Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=13.460176ms db_storage=652ns remote_storage=1.105µs web_handler=313ns query_engine=520ns scrape=451.083µs scrape_sd=17.377µs notify=817ns notify_sd=1.691µs rules=12.642612ms 2024-04-07T20:51:45.804 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:51:45.507Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-07T20:51:45.841 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-07T20:51:45.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-07T20:51:45.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-07T20:51:45.886 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-07T20:51:45.886 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-07T20:51:45.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-07T20:51:46.056 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 bash[42077]: e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 2024-04-07T20:51:46.056 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:51:45 smithi187 systemd[1]: Started Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:51:46.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:46.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-mon[25907]: Detected new or changed devices on smithi187 2024-04-07T20:51:46.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:46.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-mon[25907]: from='client.? 172.21.15.187:0/1299544715' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:46.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:46.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:45 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[29851]: Detected new or changed devices on smithi187 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[29851]: from='client.? 172.21.15.187:0/1299544715' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[24852]: Detected new or changed devices on smithi187 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[24852]: from='client.? 172.21.15.187:0/1299544715' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:46.106 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T20:51:46.107 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:45 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T20:51:46.118 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:51:47.077 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:47 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:47.077 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:47 smithi187 ceph-mon[25907]: Deploying daemon node-exporter.a on smithi177 2024-04-07T20:51:47.077 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:47 smithi187 ceph-mon[25907]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:47.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:47 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:47.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:47 smithi177 ceph-mon[29851]: Deploying daemon node-exporter.a on smithi177 2024-04-07T20:51:47.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:47 smithi177 ceph-mon[29851]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:47.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:47 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:47.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:47 smithi177 ceph-mon[24852]: Deploying daemon node-exporter.a on smithi177 2024-04-07T20:51:47.413 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:47 smithi177 ceph-mon[24852]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:51:48.446 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:48 smithi177 ceph-mon[24852]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:48.446 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:48 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/12546199' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-07T20:51:48.446 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:48 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:48.446 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:48 smithi177 ceph-mon[29851]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:48.447 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:48 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/12546199' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-07T20:51:48.447 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:48 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:48.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:48 smithi187 ceph-mon[25907]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:48.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:48 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/12546199' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-07T20:51:48.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:48 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:49.127 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":19,"active_gid":14454,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3714409812},{"type":"v1","addr":"172.21.15.177:6801","nonce":3714409812}]},"active_addr":"172.21.15.177:6801/3714409812","active_change":"2024-04-07T20:51:27.928392+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34337,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.177:8443/","prometheus":"http://172.21.15.177:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":796290409}]},{"addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3685176421}]},{"addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":448981005}]},{"addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":342982299}]}]}} 2024-04-07T20:51:49.130 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-07T20:51:49.130 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-07T20:51:49.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-07T20:51:49.553 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:51:50.392 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:50 smithi177 ceph-mon[29851]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:50.393 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:50 smithi177 ceph-mon[24852]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:50.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:50 smithi187 ceph-mon[25907]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:50.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:51:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:50.759 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:51:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:51.131 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:51:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:51.131 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 podman[45599]: 2024-04-07 20:51:51.016740526 +0000 UTC m=+3.026366218 container create 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-07T20:51:51.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.510 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 podman[45599]: 2024-04-07 20:51:51.244407041 +0000 UTC m=+3.254032734 container init 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-07T20:51:51.510 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 podman[45599]: 2024-04-07 20:51:51.247195359 +0000 UTC m=+3.256821051 container start 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-07T20:51:51.510 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.251Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-07T20:51:51.510 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.251Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-07T20:51:51.510 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.251Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.251Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=arp 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=edac 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-07T20:51:51.511 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-07T20:51:51.512 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=os 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=stat 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=time 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=uname 2024-04-07T20:51:51.513 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-07T20:51:51.514 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-07T20:51:51.514 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-07T20:51:51.514 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-07T20:51:51.514 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a[45897]: ts=2024-04-07T20:51:51.252Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-07T20:51:51.514 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 bash[45599]: 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 2024-04-07T20:51:51.514 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:51:51 smithi177 systemd[1]: Started Ceph node-exporter.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:51:51.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:51 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:51 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:51 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:51 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:51.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:51 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:52.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:52 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:52.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:52 smithi187 ceph-mon[25907]: Deploying daemon node-exporter.b on smithi187 2024-04-07T20:51:52.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:52 smithi187 ceph-mon[25907]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:52.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:52 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:52.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:52 smithi177 ceph-mon[29851]: Deploying daemon node-exporter.b on smithi187 2024-04-07T20:51:52.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:52 smithi177 ceph-mon[29851]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:52.475 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:52 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:52.475 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:52 smithi177 ceph-mon[24852]: Deploying daemon node-exporter.b on smithi187 2024-04-07T20:51:52.475 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:52 smithi177 ceph-mon[24852]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:52.806 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:52 smithi187 systemd[1]: Starting Ceph node-exporter.b for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:51:53.180 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:52 smithi187 bash[42887]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-07T20:51:53.180 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:53 smithi187 bash[42887]: Getting image source signatures 2024-04-07T20:51:53.180 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:53 smithi187 bash[42887]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-07T20:51:53.556 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:53 smithi187 bash[42887]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-07T20:51:53.556 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:53 smithi187 bash[42887]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-07T20:51:54.269 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:51:54.269 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":60,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","created":"2024-04-07T20:46:02.894482+0000","modified":"2024-04-07T20:51:27.928037+0000","last_up_change":"2024-04-07T20:51:09.781055+0000","last_in_change":"2024-04-07T20:50:51.817693+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-07T20:49:08.521467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"108f20e3-d403-4cb5-8ef2-e62d0f73fb65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6803","nonce":3585323055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6805","nonce":3585323055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6809","nonce":3585323055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6807","nonce":3585323055}]},"public_addr":"172.21.15.177:6803/3585323055","cluster_addr":"172.21.15.177:6805/3585323055","heartbeat_back_addr":"172.21.15.177:6809/3585323055","heartbeat_front_addr":"172.21.15.177:6807/3585323055","state":["exists","up"]},{"osd":1,"uuid":"30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6811","nonce":3320710587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6813","nonce":3320710587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6817","nonce":3320710587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6815","nonce":3320710587}]},"public_addr":"172.21.15.177:6811/3320710587","cluster_addr":"172.21.15.177:6813/3320710587","heartbeat_back_addr":"172.21.15.177:6817/3320710587","heartbeat_front_addr":"172.21.15.177:6815/3320710587","state":["exists","up"]},{"osd":2,"uuid":"b694bab5-4027-490c-b0cc-fe79ea645bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6819","nonce":99096110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6821","nonce":99096110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6825","nonce":99096110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6823","nonce":99096110}]},"public_addr":"172.21.15.177:6819/99096110","cluster_addr":"172.21.15.177:6821/99096110","heartbeat_back_addr":"172.21.15.177:6825/99096110","heartbeat_front_addr":"172.21.15.177:6823/99096110","state":["exists","up"]},{"osd":3,"uuid":"08cf95d7-da16-4395-ab77-f20b83859262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6827","nonce":2388937573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6829","nonce":2388937573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6833","nonce":2388937573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6831","nonce":2388937573}]},"public_addr":"172.21.15.177:6827/2388937573","cluster_addr":"172.21.15.177:6829/2388937573","heartbeat_back_addr":"172.21.15.177:6833/2388937573","heartbeat_front_addr":"172.21.15.177:6831/2388937573","state":["exists","up"]},{"osd":4,"uuid":"293dbc45-ead8-4fef-bfa0-6dfbd09b5f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6801","nonce":2244514953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6803","nonce":2244514953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6807","nonce":2244514953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6805","nonce":2244514953}]},"public_addr":"172.21.15.187:6801/2244514953","cluster_addr":"172.21.15.187:6803/2244514953","heartbeat_back_addr":"172.21.15.187:6807/2244514953","heartbeat_front_addr":"172.21.15.187:6805/2244514953","state":["exists","up"]},{"osd":5,"uuid":"a51a7901-eccb-4cc2-abc1-4b3c3d06c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6809","nonce":4121303752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6811","nonce":4121303752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6815","nonce":4121303752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6813","nonce":4121303752}]},"public_addr":"172.21.15.187:6809/4121303752","cluster_addr":"172.21.15.187:6811/4121303752","heartbeat_back_addr":"172.21.15.187:6815/4121303752","heartbeat_front_addr":"172.21.15.187:6813/4121303752","state":["exists","up"]},{"osd":6,"uuid":"bd893288-9d87-4241-ad53-ad9a8c2b69f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6817","nonce":3582202152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6819","nonce":3582202152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6823","nonce":3582202152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6821","nonce":3582202152}]},"public_addr":"172.21.15.187:6817/3582202152","cluster_addr":"172.21.15.187:6819/3582202152","heartbeat_back_addr":"172.21.15.187:6823/3582202152","heartbeat_front_addr":"172.21.15.187:6821/3582202152","state":["exists","up"]},{"osd":7,"uuid":"70a167aa-8959-4adf-864a-524538d14f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6825","nonce":1477627180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6827","nonce":1477627180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6831","nonce":1477627180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6829","nonce":1477627180}]},"public_addr":"172.21.15.187:6825/1477627180","cluster_addr":"172.21.15.187:6827/1477627180","heartbeat_back_addr":"172.21.15.187:6831/1477627180","heartbeat_front_addr":"172.21.15.187:6829/1477627180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:48:21.191935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:48:42.771552+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:04.846712+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:29.672325+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:52.446119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:50:16.546464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:50:42.572341+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:51:07.874882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:0/1769461466":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/3514106323":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/1236583948":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/851605445":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/1872289908":"2024-04-08T20:46:56.140824+0000","172.21.15.177:6800/3006842145":"2024-04-08T20:46:56.140824+0000","172.21.15.177:6800/2181256441":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/1166756411":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6801/3006842145":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/646064095":"2024-04-08T20:46:35.136972+0000","172.21.15.177:0/3203112426":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6801/2181256441":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/3172484468":"2024-04-08T20:51:27.927993+0000","172.21.15.177:6800/3045474916":"2024-04-08T20:46:35.136972+0000","172.21.15.177:0/502485757":"2024-04-08T20:51:27.927993+0000","172.21.15.177:6801/3045474916":"2024-04-08T20:46:35.136972+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-07T20:51:54.497 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:54 smithi187 ceph-mon[25907]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:54.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:54 smithi177 ceph-mon[24852]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:54.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:54 smithi177 ceph-mon[29851]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:54.806 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:54 smithi187 bash[42887]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-07T20:51:54.806 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:54 smithi187 bash[42887]: Writing manifest to image destination 2024-04-07T20:51:54.883 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-07T20:51:54.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-07T20:51:55.114 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:51:55.296 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3761614008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:55.297 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:54 smithi187 podman[42887]: 2024-04-07 20:51:54.955934859 +0000 UTC m=+2.215342847 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-07T20:51:55.297 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 podman[42887]: 2024-04-07 20:51:55.05415132 +0000 UTC m=+2.313559308 container create 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-07T20:51:55.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:55 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3761614008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:55.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:55 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3761614008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:55.556 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 podman[42887]: 2024-04-07 20:51:55.298170237 +0000 UTC m=+2.557578224 container init 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 podman[42887]: 2024-04-07 20:51:55.301615411 +0000 UTC m=+2.561023407 container start 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.306Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.306Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.306Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.306Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=arp 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-07T20:51:55.557 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=edac 2024-04-07T20:51:55.558 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-07T20:51:55.559 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=os 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=stat 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-07T20:51:55.560 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=time 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=uname 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.307Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b[42939]: ts=2024-04-07T20:51:55.308Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 bash[42887]: 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d 2024-04-07T20:51:55.561 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:51:55 smithi187 systemd[1]: Started Ceph node-exporter.b for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: Reconfiguring daemon prometheus.a on smithi187 2024-04-07T20:51:56.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:56 smithi187 ceph-mon[25907]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: Reconfiguring daemon prometheus.a on smithi187 2024-04-07T20:51:56.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[29851]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: Reconfiguring daemon prometheus.a on smithi187 2024-04-07T20:51:56.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:56 smithi177 ceph-mon[24852]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:57.659 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:57 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:51:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:58.482 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:58.482 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-mon[24852]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:58.482 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:58.482 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:58.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-mon[29851]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:58.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:58.514 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:51:58.514 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":60,"fsid":"c7f93d4c-f51f-11ee-b649-cb9ed24678a4","created":"2024-04-07T20:46:02.894482+0000","modified":"2024-04-07T20:51:27.928037+0000","last_up_change":"2024-04-07T20:51:09.781055+0000","last_in_change":"2024-04-07T20:50:51.817693+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-07T20:49:08.521467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"108f20e3-d403-4cb5-8ef2-e62d0f73fb65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6803","nonce":3585323055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6805","nonce":3585323055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6809","nonce":3585323055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3585323055},{"type":"v1","addr":"172.21.15.177:6807","nonce":3585323055}]},"public_addr":"172.21.15.177:6803/3585323055","cluster_addr":"172.21.15.177:6805/3585323055","heartbeat_back_addr":"172.21.15.177:6809/3585323055","heartbeat_front_addr":"172.21.15.177:6807/3585323055","state":["exists","up"]},{"osd":1,"uuid":"30c4b2ce-1e7d-40e6-ad00-ac5ea8cb2694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6811","nonce":3320710587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6813","nonce":3320710587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6817","nonce":3320710587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3320710587},{"type":"v1","addr":"172.21.15.177:6815","nonce":3320710587}]},"public_addr":"172.21.15.177:6811/3320710587","cluster_addr":"172.21.15.177:6813/3320710587","heartbeat_back_addr":"172.21.15.177:6817/3320710587","heartbeat_front_addr":"172.21.15.177:6815/3320710587","state":["exists","up"]},{"osd":2,"uuid":"b694bab5-4027-490c-b0cc-fe79ea645bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6819","nonce":99096110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6821","nonce":99096110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6825","nonce":99096110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":99096110},{"type":"v1","addr":"172.21.15.177:6823","nonce":99096110}]},"public_addr":"172.21.15.177:6819/99096110","cluster_addr":"172.21.15.177:6821/99096110","heartbeat_back_addr":"172.21.15.177:6825/99096110","heartbeat_front_addr":"172.21.15.177:6823/99096110","state":["exists","up"]},{"osd":3,"uuid":"08cf95d7-da16-4395-ab77-f20b83859262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6827","nonce":2388937573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6829","nonce":2388937573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6833","nonce":2388937573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2388937573},{"type":"v1","addr":"172.21.15.177:6831","nonce":2388937573}]},"public_addr":"172.21.15.177:6827/2388937573","cluster_addr":"172.21.15.177:6829/2388937573","heartbeat_back_addr":"172.21.15.177:6833/2388937573","heartbeat_front_addr":"172.21.15.177:6831/2388937573","state":["exists","up"]},{"osd":4,"uuid":"293dbc45-ead8-4fef-bfa0-6dfbd09b5f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6801","nonce":2244514953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6803","nonce":2244514953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6807","nonce":2244514953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":2244514953},{"type":"v1","addr":"172.21.15.187:6805","nonce":2244514953}]},"public_addr":"172.21.15.187:6801/2244514953","cluster_addr":"172.21.15.187:6803/2244514953","heartbeat_back_addr":"172.21.15.187:6807/2244514953","heartbeat_front_addr":"172.21.15.187:6805/2244514953","state":["exists","up"]},{"osd":5,"uuid":"a51a7901-eccb-4cc2-abc1-4b3c3d06c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6809","nonce":4121303752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6811","nonce":4121303752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6815","nonce":4121303752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4121303752},{"type":"v1","addr":"172.21.15.187:6813","nonce":4121303752}]},"public_addr":"172.21.15.187:6809/4121303752","cluster_addr":"172.21.15.187:6811/4121303752","heartbeat_back_addr":"172.21.15.187:6815/4121303752","heartbeat_front_addr":"172.21.15.187:6813/4121303752","state":["exists","up"]},{"osd":6,"uuid":"bd893288-9d87-4241-ad53-ad9a8c2b69f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6817","nonce":3582202152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6819","nonce":3582202152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6823","nonce":3582202152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":3582202152},{"type":"v1","addr":"172.21.15.187:6821","nonce":3582202152}]},"public_addr":"172.21.15.187:6817/3582202152","cluster_addr":"172.21.15.187:6819/3582202152","heartbeat_back_addr":"172.21.15.187:6823/3582202152","heartbeat_front_addr":"172.21.15.187:6821/3582202152","state":["exists","up"]},{"osd":7,"uuid":"70a167aa-8959-4adf-864a-524538d14f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6825","nonce":1477627180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6827","nonce":1477627180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6831","nonce":1477627180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1477627180},{"type":"v1","addr":"172.21.15.187:6829","nonce":1477627180}]},"public_addr":"172.21.15.187:6825/1477627180","cluster_addr":"172.21.15.187:6827/1477627180","heartbeat_back_addr":"172.21.15.187:6831/1477627180","heartbeat_front_addr":"172.21.15.187:6829/1477627180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:48:21.191935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:48:42.771552+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:04.846712+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:29.672325+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:49:52.446119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:50:16.546464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:50:42.572341+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T20:51:07.874882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:0/1769461466":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/3514106323":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/1236583948":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/851605445":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/1872289908":"2024-04-08T20:46:56.140824+0000","172.21.15.177:6800/3006842145":"2024-04-08T20:46:56.140824+0000","172.21.15.177:6800/2181256441":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/1166756411":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6801/3006842145":"2024-04-08T20:46:56.140824+0000","172.21.15.177:0/646064095":"2024-04-08T20:46:35.136972+0000","172.21.15.177:0/3203112426":"2024-04-08T20:46:35.136972+0000","172.21.15.177:6801/2181256441":"2024-04-08T20:51:27.927993+0000","172.21.15.177:0/3172484468":"2024-04-08T20:51:27.927993+0000","172.21.15.177:6800/3045474916":"2024-04-08T20:46:35.136972+0000","172.21.15.177:0/502485757":"2024-04-08T20:51:27.927993+0000","172.21.15.177:6801/3045474916":"2024-04-08T20:46:35.136972+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-07T20:51:58.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:58.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:58 smithi187 ceph-mon[25907]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:51:58.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:59.033 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:51:58] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-04-07T20:51:59.395 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:51:59 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:51:59] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:51:59.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:59 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/871104055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:59.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:51:59 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:59.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:59 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/871104055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:59.685 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:51:59 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:59.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:59 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/871104055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T20:51:59.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:51:59 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:51:59.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:51:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:51:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:52:00.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-07T20:52:00.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-07T20:52:00.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-07T20:52:00.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-07T20:52:00.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-07T20:52:00.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-07T20:52:00.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-07T20:52:00.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-07T20:52:00.425 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:00 smithi187 ceph-mon[25907]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:00.572 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:00.578 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:00.608 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:00.709 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:00 smithi177 ceph-mon[24852]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:00.709 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:00 smithi177 ceph-mon[29851]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:00.709 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:52:00] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:01.411 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:01.743 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:01.743 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:01.743 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 systemd[1]: Stopping Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:52:01.743 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 bash[43387]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-07T20:52:01.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:01 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:01.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:01 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:01.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:01 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:01.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:01 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.987Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-07T20:52:02.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:01 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[42151]: ts=2024-04-07T20:52:01.988Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-07T20:52:02.556 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:02 smithi187 podman[43398]: 2024-04-07 20:52:02.327573454 +0000 UTC m=+0.799543866 container died e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:03.056 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:02 smithi187 podman[43398]: 2024-04-07 20:52:02.748478483 +0000 UTC m=+1.220448943 container cleanup e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:03.056 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:02 smithi187 bash[43398]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-07T20:52:03.056 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:02 smithi187 bash[43464]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-07T20:52:03.310 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-mon[25907]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:03.310 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 podman[43433]: 2024-04-07 20:52:03.057732689 +0000 UTC m=+1.061892558 container remove e8c87c32b1064009034aafdd5774cb38579187dce7065c4fe4c8cf1d2672e5a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:03.311 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-07T20:52:03.311 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 systemd[1]: Stopped Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:52:03.311 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 systemd[1]: Starting Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:52:03.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:03 smithi177 ceph-mon[24852]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:03.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:03 smithi177 ceph-mon[29851]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:03.609 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 podman[43604]: 2024-04-07 20:52:03.215374793 +0000 UTC m=+0.020191445 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-07T20:52:03.609 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 podman[43604]: 2024-04-07 20:52:03.351949551 +0000 UTC m=+0.156766213 container create 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:03.609 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 podman[43604]: 2024-04-07 20:52:03.610720093 +0000 UTC m=+0.415536714 container init 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:03.867 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 podman[43604]: 2024-04-07 20:52:03.613435646 +0000 UTC m=+0.418252271 container start 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.635Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.635Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.635Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.635Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi187 (none))" 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.635Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.635Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.637Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.637Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.638Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.640Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.640Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.636µs 2024-04-07T20:52:03.868 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.640Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-07T20:52:03.869 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 bash[43604]: 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa 2024-04-07T20:52:03.869 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 systemd[1]: Started Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:52:03.869 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.868Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-07T20:52:03.869 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.869Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-07T20:52:03.869 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.869Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.639µs wal_replay_duration=228.773486ms total_replay_duration=228.811949ms 2024-04-07T20:52:04.162 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.871Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-07T20:52:04.162 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.871Z caller=main.go:947 level=info msg="TSDB started" 2024-04-07T20:52:04.162 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.871Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-07T20:52:04.162 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.886Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.441ms db_storage=747ns remote_storage=1.289µs web_handler=389ns query_engine=1.085µs scrape=355.823µs scrape_sd=22.68µs notify=839ns notify_sd=1.306µs rules=14.859457ms 2024-04-07T20:52:04.162 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:52:03 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:52:03.886Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-07T20:52:04.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:52:04] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.187:9095"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.187:9095"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.808 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.187:9095"}]: dispatch 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.187:9095"}]: dispatch 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:04.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:52:04.945 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.187:9095"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.187:9095"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:04.946 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:05.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:05 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:52:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[24852]: Deploying daemon alertmanager.a on smithi177 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-mon[29851]: Deploying daemon alertmanager.a on smithi177 2024-04-07T20:52:06.570 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:06.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:06 smithi187 ceph-mon[25907]: Deploying daemon alertmanager.a on smithi177 2024-04-07T20:52:07.212 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:06 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:52:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:07.629 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:07 smithi187 ceph-mon[25907]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-07T20:52:07.629 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:07 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:07.629 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:07 smithi187 ceph-mon[25907]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:07.629 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:07 smithi187 ceph-mon[25907]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:07.629 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:07 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:07.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[24852]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[24852]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[24852]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[29851]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[29851]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[29851]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:07.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:07 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:07.892 INFO:teuthology.orchestra.run.smithi177.stdout:34359738414 2024-04-07T20:52:07.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-07T20:52:09.224 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:08] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-04-07T20:52:09.251 INFO:teuthology.orchestra.run.smithi177.stdout:77309411366 2024-04-07T20:52:09.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-07T20:52:09.778 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:52:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:52:10.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:09 smithi187 ceph-mon[25907]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:10.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:09 smithi177 ceph-mon[29851]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:10.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:09 smithi177 ceph-mon[24852]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:10.463 INFO:teuthology.orchestra.run.smithi177.stdout:236223201293 2024-04-07T20:52:10.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-07T20:52:10.480 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:10.482 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:10.483 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:10.872 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:10.897 INFO:teuthology.orchestra.run.smithi177.stdout:111669149729 2024-04-07T20:52:10.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-07T20:52:10.909 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:11.829 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:12.250 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:11 smithi177 ceph-mon[29851]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:12.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:11 smithi177 ceph-mon[24852]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:12.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:11 smithi187 ceph-mon[25907]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:13.859 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:14.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:14 smithi177 ceph-mon[29851]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:14.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:14 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:14 smithi177 ceph-mon[24852]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:14 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:14.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:14 smithi187 ceph-mon[25907]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:14.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:14 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:16.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:16 smithi177 ceph-mon[29851]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:16.330 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:16 smithi177 ceph-mon[24852]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:16.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:16 smithi187 ceph-mon[25907]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:18.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:18 smithi177 ceph-mon[24852]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:18.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:18 smithi177 ceph-mon[29851]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:18.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:18 smithi187 ceph-mon[25907]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:19.080 INFO:teuthology.orchestra.run.smithi177.stdout:167503724570 2024-04-07T20:52:19.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-07T20:52:19.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:18] "GET /metrics HTTP/1.1" 200 191560 "" "Prometheus/2.33.4" 2024-04-07T20:52:19.556 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:52:19 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:52:19.961 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:19 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:52:19] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:20.215 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:20 smithi177 ceph-mon[29851]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:20.215 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:20 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:20.215 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:20 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:20.215 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:20 smithi177 ceph-mon[24852]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:20.215 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:20 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:20.215 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:20 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:20.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:20 smithi187 ceph-mon[25907]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:20.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:20 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:20.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:20 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:21.226 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:21 smithi187 ceph-mon[25907]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:21.226 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:21 smithi187 ceph-mon[25907]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-07T20:52:21.226 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:21 smithi187 ceph-mon[25907]: Cluster is now healthy 2024-04-07T20:52:21.313 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:21 smithi177 ceph-mon[24852]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:21.313 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:21 smithi177 ceph-mon[24852]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-07T20:52:21.314 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:21 smithi177 ceph-mon[24852]: Cluster is now healthy 2024-04-07T20:52:21.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:21 smithi177 ceph-mon[29851]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:21.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:21 smithi177 ceph-mon[29851]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-07T20:52:21.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:21 smithi177 ceph-mon[29851]: Cluster is now healthy 2024-04-07T20:52:22.050 INFO:teuthology.orchestra.run.smithi177.stdout:55834574893 2024-04-07T20:52:22.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-07T20:52:23.245 INFO:teuthology.orchestra.run.smithi177.stdout:77309411369 2024-04-07T20:52:23.468 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:23 smithi177 ceph-mon[29851]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:23.468 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:23 smithi177 ceph-mon[24852]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:23.502 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:23 smithi187 ceph-mon[25907]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:24.045 INFO:teuthology.orchestra.run.smithi177.stdout:201863462933 2024-04-07T20:52:24.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-07T20:52:24.317 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:24 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3672091129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-07T20:52:24.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:24 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3672091129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-07T20:52:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:24 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3672091129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-07T20:52:24.668 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:24 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:52:24] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:24.954 INFO:teuthology.orchestra.run.smithi177.stdout:34359738417 2024-04-07T20:52:25.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[29851]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2844154150' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[24852]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:25.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:25 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2844154150' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-07T20:52:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:25 smithi187 ceph-mon[25907]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:25 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:25.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:25 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:25.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:25 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2844154150' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-07T20:52:25.585 INFO:teuthology.orchestra.run.smithi177.stdout:236223201296 2024-04-07T20:52:26.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411369 for osd.2 2024-04-07T20:52:26.384 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:26.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:26 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/453386216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-07T20:52:26.505 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:26 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/453386216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-07T20:52:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:26 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/453386216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-07T20:52:27.452 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:27 smithi177 ceph-mon[29851]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:27.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:27 smithi177 ceph-mon[24852]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:27 smithi187 ceph-mon[25907]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:27.964 INFO:teuthology.orchestra.run.smithi177.stdout:111669149733 2024-04-07T20:52:28.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:28 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/820659784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-07T20:52:28.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:28 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/820659784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-07T20:52:28.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:28 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/820659784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:28] "GET /metrics HTTP/1.1" 200 191563 "" "Prometheus/2.33.4" 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[24852]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:52:29.165 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:52:29.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[29851]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:29.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:29.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:52:29.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:52:29.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:52:29.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:52:29.167 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201293 got 236223201296 for osd.7 2024-04-07T20:52:29.167 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:29.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-mon[25907]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:29.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:52:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:52:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:52:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:52:29.708 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149733 for osd.3 2024-04-07T20:52:29.708 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:29.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:52:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:52:30.082 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:30.083 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:30.083 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:30.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738417 for osd.0 2024-04-07T20:52:30.593 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:31.486 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:31 smithi177 ceph-mon[29851]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:31.487 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:31 smithi177 ceph-mon[24852]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:31.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:31 smithi187 ceph-mon[25907]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:32.156 INFO:teuthology.orchestra.run.smithi177.stdout:133143986209 2024-04-07T20:52:32.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-07T20:52:33.323 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:33 smithi177 ceph-mon[29851]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:33.323 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:33 smithi177 ceph-mon[24852]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:33.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:33 smithi187 ceph-mon[25907]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:34.130 INFO:teuthology.orchestra.run.smithi177.stdout:55834574896 2024-04-07T20:52:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:34 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/706748470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-07T20:52:34.262 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:34 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/706748470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-07T20:52:34.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:34 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/706748470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-07T20:52:34.592 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:35.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:35 smithi177 ceph-mon[24852]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:35.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:35 smithi177 ceph-mon[29851]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:35.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:35 smithi187 ceph-mon[25907]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:36.560 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574896 for osd.1 2024-04-07T20:52:36.561 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:37.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:37 smithi177 ceph-mon[24852]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:37.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:37 smithi177 ceph-mon[29851]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:37.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:37 smithi187 ceph-mon[25907]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:38.623 INFO:teuthology.orchestra.run.smithi177.stdout:167503724574 2024-04-07T20:52:39.153 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:38 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:38] "GET /metrics HTTP/1.1" 200 191563 "" "Prometheus/2.33.4" 2024-04-07T20:52:39.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:39 smithi187 ceph-mon[25907]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:39.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:39 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/4286832251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-07T20:52:39.480 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:39 smithi177 ceph-mon[29851]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:39.480 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:39 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/4286832251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-07T20:52:39.481 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:39 smithi177 ceph-mon[24852]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:39.481 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:39 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4286832251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-07T20:52:39.676 INFO:teuthology.orchestra.run.smithi177.stdout:133143986210 2024-04-07T20:52:39.736 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:52:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:52:40.093 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:39 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:52:39] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:40.116 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724570 got 167503724574 for osd.5 2024-04-07T20:52:40.116 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:40.344 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:40 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/196789192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-07T20:52:40.344 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:40 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:40.344 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:40 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:40.344 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:40 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/196789192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-07T20:52:40.344 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:40 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:40.344 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:40 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:40.496 INFO:teuthology.orchestra.run.smithi177.stdout:201863462937 2024-04-07T20:52:40.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:40 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/196789192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-07T20:52:40.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:40 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:40.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:40 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:41.154 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462933 got 201863462937 for osd.6 2024-04-07T20:52:41.154 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:41.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:41 smithi177 ceph-mon[29851]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:41.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:41 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3855654791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-07T20:52:41.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:41 smithi177 ceph-mon[24852]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:41.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:41 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3855654791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-07T20:52:41.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:41 smithi187 ceph-mon[25907]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:41.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:41 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3855654791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-07T20:52:41.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986209 got 133143986210 for osd.4 2024-04-07T20:52:41.635 DEBUG:teuthology.parallel:result is None 2024-04-07T20:52:41.635 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-07T20:52:41.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-07T20:52:42.008 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:41 smithi177 podman[49307]: 2024-04-07 20:52:41.838143238 +0000 UTC m=+2.909781712 container create 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:42.302 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:42.312 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 podman[49307]: 2024-04-07 20:52:42.166288236 +0000 UTC m=+3.237926711 container init 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:42.312 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 podman[49307]: 2024-04-07 20:52:42.169065551 +0000 UTC m=+3.240704039 container start 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:52:42.312 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.179Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-07T20:52:42.312 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.179Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.180Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.177 port=9094 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.182Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.206Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.207Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.209Z caller=main.go:518 msg=Listening address=:9093 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:42.209Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 bash[49307]: 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 2024-04-07T20:52:42.313 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:42 smithi177 systemd[1]: Started Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:52:43.220 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[29851]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:43.220 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[24852]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T20:52:43.221 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:43 smithi187 ceph-mon[25907]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:43 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T20:52:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:44.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T20:52:44.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-mon[24852]: Deploying daemon grafana.a on smithi187 2024-04-07T20:52:44.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:44.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T20:52:44.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-mon[29851]: Deploying daemon grafana.a on smithi187 2024-04-07T20:52:44.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:44.260 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:44.183Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000968205s 2024-04-07T20:52:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:44 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T20:52:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:44 smithi187 ceph-mon[25907]: Deploying daemon grafana.a on smithi187 2024-04-07T20:52:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:44.605 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:44 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:52:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:45.137 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:45 smithi177 ceph-mon[24852]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:45.137 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:45 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:45.138 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:45 smithi177 ceph-mon[29851]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:45.138 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:45 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:45 smithi187 ceph-mon[25907]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:45 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:46.637 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:52:46.638 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-07T20:52:47.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:47 smithi177 ceph-mon[29851]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:47.283 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:47 smithi177 ceph-mon[24852]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:47.429 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-04-07T20:52:45.661594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6295552,"data_stored":3510277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002071"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-07T20:51:28.220495+0000","last_change":"2024-04-07T20:51:14.117487+0000","last_active":"2024-04-07T20:51:28.220495+0000","last_peered":"2024-04-07T20:51:28.220495+0000","last_clean":"2024-04-07T20:51:28.220495+0000","last_became_active":"2024-04-07T20:51:14.117081+0000","last_became_peered":"2024-04-07T20:51:14.117081+0000","last_unstale":"2024-04-07T20:51:28.220495+0000","last_undegraded":"2024-04-07T20:51:28.220495+0000","last_fullsized":"2024-04-07T20:51:28.220495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-07T20:49:08.571632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-07T20:49:08.571632+0000","last_clean_scrub_stamp":"2024-04-07T20:49:08.571632+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T23:53:27.352370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56399999999999995}]},{"osd":1,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55700000000000005}]},{"osd":3,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.46000000000000002}]},{"osd":5,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.439},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47499999999999998}]},{"osd":6,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.497}]}]},{"osd":6,"up_from":47,"seq":201863462938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Apr 7 20:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41099999999999998}]},{"osd":2,"last update":"Sun Apr 7 20:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.434}]},{"osd":3,"last update":"Sun Apr 7 20:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.49199999999999999}]},{"osd":4,"last update":"Sun Apr 7 20:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.443},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.45200000000000001}]},{"osd":5,"last update":"Sun Apr 7 20:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.47399999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.26400000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.33800000000000002},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.501,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.46700000000000003}]},{"osd":2,"last update":"Sun Apr 7 20:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.36499999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55300000000000005,"5min":0.621,"15min":0.621},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.38800000000000001,"15min":0.38400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.50700000000000001,"5min":0.623,"15min":0.623},"last":0.318}]},{"osd":3,"last update":"Sun Apr 7 20:52:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54400000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.432},{"interface":"front","average":{"1min":0.443,"5min":0.433,"15min":0.439},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.42499999999999999}]},{"osd":4,"last update":"Sun Apr 7 20:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.80700000000000005}]},{"osd":5,"last update":"Sun Apr 7 20:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72899999999999998}]},{"osd":6,"last update":"Sun Apr 7 20:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.77000000000000002}]},{"osd":7,"last update":"Sun Apr 7 20:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.60299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 7 20:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.28499999999999998}]},{"osd":2,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53500000000000003,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.222}]},{"osd":3,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48799999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.25},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.44800000000000001,"15min":0.46400000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.51100000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Sun Apr 7 20:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80500000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.37}]},{"osd":5,"last update":"Sun Apr 7 20:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67700000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68400000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.38100000000000001}]},{"osd":6,"last update":"Sun Apr 7 20:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.41999999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":239784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.34599999999999997},"min":{"1min":0.23999999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.315},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.35299999999999998},"min":{"1min":0.26800000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.36599999999999999}]},{"osd":1,"last update":"Sun Apr 7 20:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.374,"15min":0.35499999999999998},"min":{"1min":0.23999999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.38},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.38100000000000001,"15min":0.36199999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33100000000000002}]},{"osd":3,"last update":"Sun Apr 7 20:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.441,"15min":0.435},"min":{"1min":0.30599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56999999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59099999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Apr 7 20:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.497},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.58799999999999997}]},{"osd":5,"last update":"Sun Apr 7 20:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60099999999999998}]},{"osd":6,"last update":"Sun Apr 7 20:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47999999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55400000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38200000000000001,"15min":0.39800000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.50600000000000001}]},{"osd":1,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47599999999999998}]},{"osd":2,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.40699999999999997,"15min":0.40899999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.45000000000000001}]},{"osd":4,"last update":"Sun Apr 7 20:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53400000000000003,"15min":0.52500000000000002},"min":{"1min":0.44500000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.61299999999999999}]},{"osd":5,"last update":"Sun Apr 7 20:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Sun Apr 7 20:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.63400000000000001}]},{"osd":7,"last update":"Sun Apr 7 20:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.622},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56299999999999994}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.376}]},{"osd":1,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.503,"15min":0.503},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65500000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.39100000000000001}]},{"osd":2,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52900000000000003}]},{"osd":3,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52000000000000002,"15min":0.51800000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46000000000000002}]},{"osd":5,"last update":"Sun Apr 7 20:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.307,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57599999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.34200000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Sun Apr 7 20:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.47399999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.438},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.628}]}]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.628,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.39000000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64600000000000002,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.435}]},{"osd":1,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.68700000000000006,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.49099999999999999}]},{"osd":2,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54900000000000004,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68100000000000005,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.55800000000000005,"5min":0.754,"15min":0.754},"last":0.57199999999999995}]},{"osd":3,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.64100000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.497,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.60499999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52300000000000002}]},{"osd":4,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56699999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.59099999999999997}]},{"osd":6,"last update":"Sun Apr 7 20:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.61199999999999999}]},{"osd":7,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-07T20:52:47.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-07T20:52:47.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:47 smithi187 ceph-mon[25907]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:47.649 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:48.376 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:48 smithi177 ceph-mon[24852]: from='client.24529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:52:48.376 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:52:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:48.376 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:48 smithi177 ceph-mon[29851]: from='client.24529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:52:48.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:48 smithi187 ceph-mon[25907]: from='client.24529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:52:49.249 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:48] "GET /metrics HTTP/1.1" 200 191567 "" "Prometheus/2.33.4" 2024-04-07T20:52:49.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:49 smithi177 ceph-mon[29851]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:49.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:49 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:49.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:49 smithi177 ceph-mon[24852]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:49.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:49 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:49.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:49 smithi187 ceph-mon[25907]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:49.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:49 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:49.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:52:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:52:50.565 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:52:50.566 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-07T20:52:51.334 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-04-07T20:52:49.662376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6295552,"data_stored":3510277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002215"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-07T20:51:28.220495+0000","last_change":"2024-04-07T20:51:14.117487+0000","last_active":"2024-04-07T20:51:28.220495+0000","last_peered":"2024-04-07T20:51:28.220495+0000","last_clean":"2024-04-07T20:51:28.220495+0000","last_became_active":"2024-04-07T20:51:14.117081+0000","last_became_peered":"2024-04-07T20:51:14.117081+0000","last_unstale":"2024-04-07T20:51:28.220495+0000","last_undegraded":"2024-04-07T20:51:28.220495+0000","last_fullsized":"2024-04-07T20:51:28.220495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-07T20:49:08.571632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-07T20:49:08.571632+0000","last_clean_scrub_stamp":"2024-04-07T20:49:08.571632+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T23:53:27.352370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.438},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47899999999999998}]},{"osd":3,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.39000000000000001}]},{"osd":5,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.373}]},{"osd":6,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.50700000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72699999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46800000000000003}]},{"osd":1,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.54600000000000004,"15min":0.55700000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.60199999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.436},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.53200000000000003,"15min":0.54500000000000004},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.432}]},{"osd":2,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76700000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.495},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.5}]},{"osd":3,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56299999999999994},"min":{"1min":0.33900000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59099999999999997,"15min":0.59499999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.315,"5min":0.224,"15min":0.224},"max":{"1min":0.70999999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.308,"5min":0.246,"15min":0.246},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39300000000000002}]},{"osd":5,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45100000000000001},"min":{"1min":0.30399999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47599999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67800000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45900000000000002}]},{"osd":7,"last update":"Sun Apr 7 20:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:51:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.26400000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.33800000000000002},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.501,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Apr 7 20:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.36499999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55300000000000005,"5min":0.621,"15min":0.621},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.38800000000000001,"15min":0.38400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.50700000000000001,"5min":0.623,"15min":0.623},"last":0.42999999999999999}]},{"osd":3,"last update":"Sun Apr 7 20:52:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54400000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.433,"15min":0.439},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.40999999999999998}]},{"osd":4,"last update":"Sun Apr 7 20:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.505}]},{"osd":5,"last update":"Sun Apr 7 20:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54100000000000004}]},{"osd":6,"last update":"Sun Apr 7 20:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80200000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52200000000000002}]},{"osd":7,"last update":"Sun Apr 7 20:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 7 20:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.316},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53500000000000003,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.379},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.36499999999999999}]},{"osd":3,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.48799999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.44800000000000001,"15min":0.46400000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.51100000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.39900000000000002}]},{"osd":4,"last update":"Sun Apr 7 20:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80500000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42899999999999999}]},{"osd":5,"last update":"Sun Apr 7 20:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67700000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68400000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47899999999999998}]},{"osd":6,"last update":"Sun Apr 7 20:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.44}]},{"osd":7,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.47199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":239784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.34599999999999997},"min":{"1min":0.23999999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.375},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.35299999999999998},"min":{"1min":0.26800000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.31900000000000001}]},{"osd":1,"last update":"Sun Apr 7 20:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.374,"15min":0.35499999999999998},"min":{"1min":0.23999999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.38100000000000001,"15min":0.36199999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.34200000000000003}]},{"osd":3,"last update":"Sun Apr 7 20:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.441,"15min":0.435},"min":{"1min":0.30599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56999999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59099999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.30399999999999999}]},{"osd":4,"last update":"Sun Apr 7 20:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.497}]},{"osd":5,"last update":"Sun Apr 7 20:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.57499999999999996}]},{"osd":6,"last update":"Sun Apr 7 20:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.47899999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51800000000000002}]},{"osd":7,"last update":"Sun Apr 7 20:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.439},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50600000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38200000000000001,"15min":0.39800000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.316},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.378},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Sun Apr 7 20:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.40699999999999997,"15min":0.40899999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42099999999999999}]},{"osd":4,"last update":"Sun Apr 7 20:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53400000000000003,"15min":0.52500000000000002},"min":{"1min":0.44500000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.46800000000000003}]},{"osd":5,"last update":"Sun Apr 7 20:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.48899999999999999}]},{"osd":6,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.58799999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.441},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.45600000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.503,"15min":0.501},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.50700000000000001}]},{"osd":1,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.503,"15min":0.503},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65500000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.60399999999999998}]},{"osd":3,"last update":"Sun Apr 7 20:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52000000000000002,"15min":0.51800000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.61899999999999999}]},{"osd":5,"last update":"Sun Apr 7 20:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.307,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57599999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.34200000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.39200000000000002}]},{"osd":6,"last update":"Sun Apr 7 20:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.436,"15min":0.42199999999999999},"min":{"1min":0.32400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.436,"15min":0.42499999999999999},"min":{"1min":0.34899999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41199999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.439}]}]},{"osd":5,"up_from":39,"seq":167503724577,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.628,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.442},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.39000000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64600000000000002,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.51900000000000002}]},{"osd":1,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.68700000000000006,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54900000000000004,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68100000000000005,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.55800000000000005,"5min":0.754,"15min":0.754},"last":0.39400000000000002}]},{"osd":3,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.64100000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.60499999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.41799999999999998}]},{"osd":4,"last update":"Sun Apr 7 20:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56699999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.626}]},{"osd":6,"last update":"Sun Apr 7 20:52:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45600000000000002,"15min":0.45000000000000001},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66100000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.60599999999999998}]},{"osd":7,"last update":"Sun Apr 7 20:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-07T20:52:51.335 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-07T20:52:51.335 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-07T20:52:51.335 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-07T20:52:51.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-07T20:52:51.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:51 smithi177 ceph-mon[24852]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:51 smithi177 ceph-mon[29851]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:51 smithi187 ceph-mon[25907]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:52.251 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:52 smithi187 ceph-mon[25907]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:52:52.263 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-07T20:52:52.390 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:52 smithi177 ceph-mon[29851]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:52:52.390 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:52 smithi177 ceph-mon[24852]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T20:52:52.390 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:52:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:52:52.184Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002442002s 2024-04-07T20:52:53.311 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:53 smithi187 ceph-mon[25907]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:53.486 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:53 smithi177 ceph-mon[29851]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:53.486 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:53 smithi177 ceph-mon[24852]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:54.927 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:54 smithi187 systemd[1]: Starting Ceph grafana.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:52:55.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:54 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:52:54] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-07T20:52:55.153 INFO:teuthology.orchestra.run.smithi177.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-07T20:52:55.187 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-mon[25907]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:55.187 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:55.187 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:55.187 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:54 smithi187 podman[44684]: 2024-04-07 20:52:54.830356927 +0000 UTC m=+0.017360031 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:54 smithi187 podman[44684]: 2024-04-07 20:52:54.930883086 +0000 UTC m=+0.117886245 container create 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, release=236.1648460182, io.buildah.version=1.24.2, version=8.5) 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 podman[44684]: 2024-04-07 20:52:55.158072421 +0000 UTC m=+0.345075522 container init 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, description=Ceph Grafana Container, name=ubi8, com.redhat.component=ubi8-container, vcs-type=git, io.buildah.version=1.24.2, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 podman[44684]: 2024-04-07 20:52:55.160837116 +0000 UTC m=+0.347840217 container start 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.component=ubi8-container, architecture=x86_64, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, io.openshift.tags=base rhel8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, version=8.5, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2) 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-07T20:52:55.188 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="App mode production" logger=settings 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-07T20:52:55.189 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-07T20:52:55.421 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:55 smithi177 ceph-mon[29851]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:55.421 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:55 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:55.421 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:55 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:55.422 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:55 smithi177 ceph-mon[24852]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:55.422 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:55 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:55.422 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:55 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-07T20:52:55.439 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-07T20:52:55.440 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-07T20:52:55.441 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-07T20:52:55.443 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-07T20:52:55.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 bash[44684]: 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 systemd[1]: Started Ceph grafana.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:52:55.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-07T20:52:55.446 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-07T20:52:55.447 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-07T20:52:55.448 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-07T20:52:55.449 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-07T20:52:55.450 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-07T20:52:55.451 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-07T20:52:55.452 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-07T20:52:55.453 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-07T20:52:55.454 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-07T20:52:55.454 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-07T20:52:55.454 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-07T20:52:55.454 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-07T20:52:55.691 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-07T20:52:55.692 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-07T20:52:55.693 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-07T20:52:55.694 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-07T20:52:55.695 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-07T20:52:55.695 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-07T20:52:55.696 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-07T20:52:55.697 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-07T20:52:55.698 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-07T20:52:55.699 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-07T20:52:55.700 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-07T20:52:55.701 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-07T20:52:55.702 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-07T20:52:55.703 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-07T20:52:55.704 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-07T20:52:55.705 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-07T20:52:55.706 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-07T20:52:55.707 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-07T20:52:55.878 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-07T20:52:55.878 INFO:tasks.cephadm:Setup complete, yielding 2024-04-07T20:52:55.879 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-07T20:52:55.889 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi177.front.sepia.ceph.com 2024-04-07T20:52:55.889 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-07T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=519.215005ms 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-07T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-07T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-07T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-07T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:52:55 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:52:55+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3656704195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3656704195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:56.459 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:56.460 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:56 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:56.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:56 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3656704195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-07T20:52:56.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:56 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:56.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:56 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:52:56.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:56 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:52:56.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:56 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:52:56.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:56 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[29851]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-07T20:52:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[29851]: Reconfiguring daemon alertmanager.a on smithi177 2024-04-07T20:52:57.150 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[29851]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:57.150 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:57.150 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[24852]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-07T20:52:57.151 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[24852]: Reconfiguring daemon alertmanager.a on smithi177 2024-04-07T20:52:57.151 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:57 smithi177 ceph-mon[24852]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:57.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:57 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:52:57.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:57 smithi187 ceph-mon[25907]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-07T20:52:57.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:57 smithi187 ceph-mon[25907]: Reconfiguring daemon alertmanager.a on smithi177 2024-04-07T20:52:57.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:57 smithi187 ceph-mon[25907]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:59.149 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:58] "GET /metrics HTTP/1.1" 200 191561 "" "Prometheus/2.33.4" 2024-04-07T20:52:59.149 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:52:59] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:52:59.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:59 smithi187 ceph-mon[25907]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:59 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:52:59 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:59.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-mon[29851]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:59.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:59.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:59.505 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-mon[24852]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:52:59.505 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:59.506 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:52:59 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:52:59.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:52:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:52:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:53:01.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:01 smithi177 ceph-mon[29851]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:01.433 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:01 smithi177 ceph-mon[24852]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:01.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:01 smithi187 ceph-mon[25907]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:01.733 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:01 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:01.733 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:01 smithi177 systemd[1]: Stopping Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:53:01.733 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:01 smithi177 bash[50828]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-07T20:53:02.008 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:01 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[49466]: level=info ts=2024-04-07T20:53:01.862Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-07T20:53:02.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:01 smithi177 podman[50840]: 2024-04-07 20:53:01.883309731 +0000 UTC m=+0.361427304 container died 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:02.638 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:02 smithi177 podman[50840]: 2024-04-07 20:53:02.347116341 +0000 UTC m=+0.825233922 container cleanup 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:02.639 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:02 smithi177 bash[50840]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-07T20:53:02.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:02.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:02.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[29851]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:02.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' 2024-04-07T20:53:02.992 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:02.993 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:02.993 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[24852]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:02.993 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:02 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' 2024-04-07T20:53:03.019 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-07T20:53:03.028 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi177.front.sepia.ceph.com 2024-04-07T20:53:03.029 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-07T20:53:03.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:02 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:03.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:02 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:03.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:02 smithi187 ceph-mon[25907]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:03.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:02 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' 2024-04-07T20:53:03.259 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:02 smithi177 bash[50904]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-07T20:53:03.259 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:03 smithi177 podman[50876]: 2024-04-07 20:53:03.202485831 +0000 UTC m=+1.318638175 container remove 3dabc64b3445261320b6a51a9ea2d617be094ac56d52d32b759a2506eb6af926 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:03.568 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:03 smithi177 podman[50876]: 2024-04-07 20:53:03.303367062 +0000 UTC m=+1.419519406 volume remove 49d64d6b5770599fd47a7f45f9520f726a72cfad7970b6d185ad9bf428531a6a 2024-04-07T20:53:03.569 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:03 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-07T20:53:03.569 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:03 smithi177 systemd[1]: Stopped Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:53:03.569 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:03 smithi177 systemd[1]: Starting Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:53:04.483 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 podman[51022]: 2024-04-07 20:53:04.057800865 +0000 UTC m=+0.350684851 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-07T20:53:04.483 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 podman[51022]: 2024-04-07 20:53:04.158375055 +0000 UTC m=+0.451259090 volume create a4270123e7b9eb673eb9f1250213b81c80c8239e4ce4a7bd8747aed662a86ea2 2024-04-07T20:53:04.483 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 podman[51022]: 2024-04-07 20:53:04.259019788 +0000 UTC m=+0.551903816 container create bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:05.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-mon[29851]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:05.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-mon[24852]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 podman[51022]: 2024-04-07 20:53:04.595108325 +0000 UTC m=+0.887992312 container init bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 podman[51022]: 2024-04-07 20:53:04.598034015 +0000 UTC m=+0.890918005 container start bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.609Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.609Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.611Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.177 port=9094 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.612Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.637Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.637Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.639Z caller=main.go:518 msg=Listening address=:9093 2024-04-07T20:53:05.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:04.639Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-07T20:53:05.010 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 bash[51022]: bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e 2024-04-07T20:53:05.010 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:04 smithi177 systemd[1]: Started Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:53:05.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:04 smithi187 ceph-mon[25907]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:06.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:05 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:06.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:05 smithi177 ceph-mon[24852]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-07T20:53:06.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:05 smithi177 ceph-mon[24852]: Reconfiguring daemon prometheus.a on smithi187 2024-04-07T20:53:06.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:05 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:06.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:05 smithi177 ceph-mon[29851]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-07T20:53:06.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:05 smithi177 ceph-mon[29851]: Reconfiguring daemon prometheus.a on smithi187 2024-04-07T20:53:06.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:05 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:06.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:05 smithi187 ceph-mon[25907]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-07T20:53:06.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:05 smithi187 ceph-mon[25907]: Reconfiguring daemon prometheus.a on smithi187 2024-04-07T20:53:06.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:06 smithi187 ceph-mon[25907]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:06.964 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:06 smithi177 ceph-mon[24852]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:06.965 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:06 smithi177 ceph-mon[29851]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:06.965 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:06 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:06.612Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000034341s 2024-04-07T20:53:07.773 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:53:07.773 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "8eaeb272-50a6-40b3-92e3-252be4b85b1f", 2024-04-07T20:53:07.774 INFO:teuthology.orchestra.run.smithi177.stdout: "name": "r", 2024-04-07T20:53:07.774 INFO:teuthology.orchestra.run.smithi177.stdout: "current_period": "be889bcc-bd43-4b12-93ae-a6807ce30e49", 2024-04-07T20:53:07.774 INFO:teuthology.orchestra.run.smithi177.stdout: "epoch": 1 2024-04-07T20:53:07.774 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:53:08.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:07 smithi177 ceph-mon[24852]: osdmap e61: 8 total, 8 up, 8 in 2024-04-07T20:53:08.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:07 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/123977650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-07T20:53:08.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:07 smithi177 ceph-mon[29851]: osdmap e61: 8 total, 8 up, 8 in 2024-04-07T20:53:08.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:07 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/123977650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-07T20:53:08.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:07 smithi187 ceph-mon[25907]: osdmap e61: 8 total, 8 up, 8 in 2024-04-07T20:53:08.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:07 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/123977650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-07T20:53:08.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:07 smithi187 systemd[1]: Stopping Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:53:08.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:07 smithi187 bash[45504]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-07T20:53:08.345 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.062Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-07T20:53:08.345 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.062Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.062Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.062Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.062Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.062Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.063Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.063Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.063Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.063Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.063Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[43640]: ts=2024-04-07T20:53:08.063Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-07T20:53:08.346 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 podman[45517]: 2024-04-07 20:53:08.07294359 +0000 UTC m=+0.136905876 container died 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:08.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-07T20:53:08.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 podman[45517]: 2024-04-07 20:53:08.45417623 +0000 UTC m=+0.518138558 container cleanup 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:08.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 bash[45517]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-07T20:53:08.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 bash[45543]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-07T20:53:09.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:08 smithi177 ceph-mon[29851]: pgmap v58: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:09.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:08 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/123977650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-07T20:53:09.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:08 smithi177 ceph-mon[29851]: osdmap e62: 8 total, 8 up, 8 in 2024-04-07T20:53:09.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:08 smithi177 ceph-mon[24852]: pgmap v58: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:09.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:08 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/123977650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-07T20:53:09.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:08 smithi177 ceph-mon[24852]: osdmap e62: 8 total, 8 up, 8 in 2024-04-07T20:53:09.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-mon[25907]: pgmap v58: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:09.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/123977650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-07T20:53:09.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:08 smithi187 ceph-mon[25907]: osdmap e62: 8 total, 8 up, 8 in 2024-04-07T20:53:09.056 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 podman[45531]: 2024-04-07 20:53:08.696586452 +0000 UTC m=+0.625566779 container remove 5d2aa19c680a16213bb672e7a35c6d42f47026096f9875beb811323cf08cc8fa (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:09.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-07T20:53:09.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 systemd[1]: Stopped Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:53:09.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 systemd[1]: Starting Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:53:09.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 podman[45615]: 2024-04-07 20:53:08.835635198 +0000 UTC m=+0.019008439 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-07T20:53:09.057 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:08 smithi187 podman[45615]: 2024-04-07 20:53:08.939304897 +0000 UTC m=+0.122678170 container create ac118aa18f679baa6a057190b30bb75e6b1059693017e3920ccbbce181204fe9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 podman[45615]: 2024-04-07 20:53:09.200160014 +0000 UTC m=+0.383533257 container init ac118aa18f679baa6a057190b30bb75e6b1059693017e3920ccbbce181204fe9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 podman[45615]: 2024-04-07 20:53:09.202895493 +0000 UTC m=+0.386268725 container start ac118aa18f679baa6a057190b30bb75e6b1059693017e3920ccbbce181204fe9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.225Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.225Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.225Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.225Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi187 (none))" 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.225Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.225Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.227Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-07T20:53:09.434 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.228Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.229Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.231Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.231Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.55µs 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.231Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 bash[45615]: ac118aa18f679baa6a057190b30bb75e6b1059693017e3920ccbbce181204fe9 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 systemd[1]: Started Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:53:09.435 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.434Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-07T20:53:09.759 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.584Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-07T20:53:09.759 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.584Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-07T20:53:09.759 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.584Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.46µs wal_replay_duration=353.03628ms total_replay_duration=353.080072ms 2024-04-07T20:53:09.759 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.587Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-07T20:53:09.759 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.587Z caller=main.go:947 level=info msg="TSDB started" 2024-04-07T20:53:09.760 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.587Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-07T20:53:09.760 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.604Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.894562ms db_storage=737ns remote_storage=1.317µs web_handler=301ns query_engine=618ns scrape=426.303µs scrape_sd=25.344µs notify=30.958µs notify_sd=8.093µs rules=16.182302ms 2024-04-07T20:53:09.760 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:53:09.604Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: osdmap e63: 8 total, 8 up, 8 in 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.177:9093"}]: dispatch 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:53:10.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.187:3000"}]: dispatch 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: osdmap e63: 8 total, 8 up, 8 in 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.031 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.177:9093"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.187:3000"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:10.032 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: osdmap e63: 8 total, 8 up, 8 in 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.177:9093"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.187:3000"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:10.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:10.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "1c78eaa7-d145-4dec-a58a-6b3a8af26388", 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "name": "default", 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "api_name": "default", 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "is_master": "true", 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "endpoints": [], 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "hostnames": [], 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "hostnames_s3website": [], 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "master_zone": "", 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "zones": [], 2024-04-07T20:53:10.718 INFO:teuthology.orchestra.run.smithi177.stdout: "placement_targets": [], 2024-04-07T20:53:10.719 INFO:teuthology.orchestra.run.smithi177.stdout: "default_placement": "", 2024-04-07T20:53:10.719 INFO:teuthology.orchestra.run.smithi177.stdout: "realm_id": "8eaeb272-50a6-40b3-92e3-252be4b85b1f", 2024-04-07T20:53:10.719 INFO:teuthology.orchestra.run.smithi177.stdout: "sync_policy": { 2024-04-07T20:53:10.719 INFO:teuthology.orchestra.run.smithi177.stdout: "groups": [] 2024-04-07T20:53:10.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:10.719 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.177:9093"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.187:3000"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[24852]: pgmap v61: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.177:9093"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:53:11.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T20:53:11.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.187:3000"}]: dispatch 2024-04-07T20:53:11.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:10 smithi177 ceph-mon[29851]: pgmap v61: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-07T20:53:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:10 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T20:53:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:10 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.177:9093"}]: dispatch 2024-04-07T20:53:11.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:10 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T20:53:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:10 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T20:53:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:10 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.187:3000"}]: dispatch 2024-04-07T20:53:11.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:10 smithi187 ceph-mon[25907]: pgmap v61: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-07T20:53:11.414 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-07T20:53:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:13 smithi177 ceph-mon[24852]: pgmap v62: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-07T20:53:13.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:53:13] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:13.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:13 smithi177 ceph-mon[29851]: pgmap v62: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-07T20:53:13.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:13 smithi187 ceph-mon[25907]: pgmap v62: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "f8ebe263-ca7a-46ec-af8e-7a1dc3b94ad7", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "name": "z", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "domain_root": "z.rgw.meta:root", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "control_pool": "z.rgw.control", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "log_pool": "z.rgw.log", 2024-04-07T20:53:14.112 INFO:teuthology.orchestra.run.smithi177.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "otp_pool": "z.rgw.otp", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "system_key": { 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "access_key": "", 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "secret_key": "" 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: "placement_pools": [ 2024-04-07T20:53:14.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "default-placement", 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "val": { 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "storage_classes": { 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "STANDARD": { 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "index_type": 0 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "realm_id": "8eaeb272-50a6-40b3-92e3-252be4b85b1f", 2024-04-07T20:53:14.114 INFO:teuthology.orchestra.run.smithi177.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-07T20:53:14.115 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:53:14.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:14 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:14.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:14 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:14 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:14.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:14 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:14.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:14 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:14.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:14 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:14.758 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:53:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:53:14.616Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00414142s 2024-04-07T20:53:14.849 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-07T20:53:15.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:15 smithi177 ceph-mon[29851]: pgmap v63: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-07T20:53:15.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:15 smithi177 ceph-mon[24852]: pgmap v63: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-07T20:53:15.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:15 smithi187 ceph-mon[25907]: pgmap v63: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-07T20:53:16.293 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:16 smithi177 ceph-mon[29851]: pgmap v64: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-07T20:53:16.293 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:16 smithi177 ceph-mon[24852]: pgmap v64: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-07T20:53:16.343 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:16 smithi187 ceph-mon[25907]: pgmap v64: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-07T20:53:19.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[24852]: pgmap v65: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-07T20:53:19.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[24852]: osdmap e64: 8 total, 8 up, 8 in 2024-04-07T20:53:19.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2607046004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-07T20:53:19.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-07T20:53:19.260 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:18] "GET /metrics HTTP/1.1" 200 192915 "" "Prometheus/2.33.4" 2024-04-07T20:53:19.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[29851]: pgmap v65: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-07T20:53:19.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[29851]: osdmap e64: 8 total, 8 up, 8 in 2024-04-07T20:53:19.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2607046004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-07T20:53:19.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:19 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-07T20:53:19.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:19 smithi187 ceph-mon[25907]: pgmap v65: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-07T20:53:19.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:19 smithi187 ceph-mon[25907]: osdmap e64: 8 total, 8 up, 8 in 2024-04-07T20:53:19.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:19 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2607046004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-07T20:53:19.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:19 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-07T20:53:19.741 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:53:19 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:53:20.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:20 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-07T20:53:20.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:20 smithi187 ceph-mon[25907]: osdmap e65: 8 total, 8 up, 8 in 2024-04-07T20:53:20.450 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:20 smithi187 ceph-mon[25907]: pgmap v68: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-07T20:53:20.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:20 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-07T20:53:20.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:20 smithi177 ceph-mon[29851]: osdmap e65: 8 total, 8 up, 8 in 2024-04-07T20:53:20.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:20 smithi177 ceph-mon[29851]: pgmap v68: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-07T20:53:20.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:20 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-07T20:53:20.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:20 smithi177 ceph-mon[24852]: osdmap e65: 8 total, 8 up, 8 in 2024-04-07T20:53:20.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:20 smithi177 ceph-mon[24852]: pgmap v68: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-07T20:53:21.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-mon[29851]: osdmap e66: 8 total, 8 up, 8 in 2024-04-07T20:53:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2607046004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-07T20:53:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-07T20:53:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-mon[24852]: osdmap e66: 8 total, 8 up, 8 in 2024-04-07T20:53:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2607046004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-07T20:53:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-07T20:53:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:21 smithi187 ceph-mon[25907]: osdmap e66: 8 total, 8 up, 8 in 2024-04-07T20:53:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:21 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2607046004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-07T20:53:21.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:21 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-07T20:53:22.010 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:21 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:21] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:22.373 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-07T20:53:22.373 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[24852]: osdmap e67: 8 total, 8 up, 8 in 2024-04-07T20:53:22.373 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[24852]: pgmap v71: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-07T20:53:22.373 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:22.374 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:22.374 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-07T20:53:22.374 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[29851]: osdmap e67: 8 total, 8 up, 8 in 2024-04-07T20:53:22.374 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[29851]: pgmap v71: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-07T20:53:22.374 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:22.374 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:22 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-07T20:53:22.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:22 smithi187 ceph-mon[25907]: osdmap e67: 8 total, 8 up, 8 in 2024-04-07T20:53:22.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:22 smithi187 ceph-mon[25907]: pgmap v71: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-07T20:53:22.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:22 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:22.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:22 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:22.669 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:53:22] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[24852]: osdmap e68: 8 total, 8 up, 8 in 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[29851]: osdmap e68: 8 total, 8 up, 8 in 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:23 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:23.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:23 smithi187 ceph-mon[25907]: osdmap e68: 8 total, 8 up, 8 in 2024-04-07T20:53:23.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:23 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-07T20:53:23.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:23 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:23.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:23 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-07T20:53:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[24852]: osdmap e69: 8 total, 8 up, 8 in 2024-04-07T20:53:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-07T20:53:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[24852]: pgmap v74: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:24.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-07T20:53:24.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[29851]: osdmap e69: 8 total, 8 up, 8 in 2024-04-07T20:53:24.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-07T20:53:24.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:24 smithi177 ceph-mon[29851]: pgmap v74: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:24 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-07T20:53:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:24 smithi187 ceph-mon[25907]: osdmap e69: 8 total, 8 up, 8 in 2024-04-07T20:53:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:24 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-07T20:53:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:24 smithi187 ceph-mon[25907]: pgmap v74: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:25.287 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:53:25.287 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "9b89e9b5-6f3b-470f-a23f-504041e2ee88", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "epoch": 1, 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "predecessor_uuid": "be889bcc-bd43-4b12-93ae-a6807ce30e49", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "sync_status": [], 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "period_map": { 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "9b89e9b5-6f3b-470f-a23f-504041e2ee88", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "zonegroups": [ 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "1c78eaa7-d145-4dec-a58a-6b3a8af26388", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "name": "default", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "api_name": "default", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "is_master": "true", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "endpoints": [], 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "hostnames": [], 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "hostnames_s3website": [], 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "master_zone": "f8ebe263-ca7a-46ec-af8e-7a1dc3b94ad7", 2024-04-07T20:53:25.288 INFO:teuthology.orchestra.run.smithi177.stdout: "zones": [ 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "id": "f8ebe263-ca7a-46ec-af8e-7a1dc3b94ad7", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "name": "z", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "endpoints": [], 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "log_meta": "false", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "log_data": "false", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "bucket_index_max_shards": 11, 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "read_only": "false", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "tier_type": "", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "sync_from_all": "true", 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "sync_from": [], 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "redirect_zone": "" 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-07T20:53:25.289 INFO:teuthology.orchestra.run.smithi177.stdout: "placement_targets": [ 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "name": "default-placement", 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "tags": [], 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "storage_classes": [ 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "STANDARD" 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "default_placement": "default-placement", 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "realm_id": "8eaeb272-50a6-40b3-92e3-252be4b85b1f", 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "sync_policy": { 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: "groups": [] 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:25.290 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "short_zone_ids": [ 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "f8ebe263-ca7a-46ec-af8e-7a1dc3b94ad7", 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 2983279538 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "master_zonegroup": "1c78eaa7-d145-4dec-a58a-6b3a8af26388", 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "master_zone": "f8ebe263-ca7a-46ec-af8e-7a1dc3b94ad7", 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "period_config": { 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "bucket_quota": { 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "enabled": false, 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "check_on_raw": false, 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "max_size": -1, 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "max_size_kb": 0, 2024-04-07T20:53:25.291 INFO:teuthology.orchestra.run.smithi177.stdout: "max_objects": -1 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "user_quota": { 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "enabled": false, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "check_on_raw": false, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_size": -1, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_size_kb": 0, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_objects": -1 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "user_ratelimit": { 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_read_ops": 0, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_write_ops": 0, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_read_bytes": 0, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "max_write_bytes": 0, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "enabled": false 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:25.292 INFO:teuthology.orchestra.run.smithi177.stdout: "bucket_ratelimit": { 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_read_ops": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_write_ops": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_read_bytes": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_write_bytes": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "enabled": false 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "anonymous_ratelimit": { 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_read_ops": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_write_ops": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_read_bytes": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "max_write_bytes": 0, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "enabled": false 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:53:25.293 INFO:teuthology.orchestra.run.smithi177.stdout: "realm_id": "8eaeb272-50a6-40b3-92e3-252be4b85b1f", 2024-04-07T20:53:25.294 INFO:teuthology.orchestra.run.smithi177.stdout: "realm_name": "r", 2024-04-07T20:53:25.294 INFO:teuthology.orchestra.run.smithi177.stdout: "realm_epoch": 2 2024-04-07T20:53:25.294 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:53:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:25 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-07T20:53:25.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:25 smithi177 ceph-mon[24852]: osdmap e70: 8 total, 8 up, 8 in 2024-04-07T20:53:25.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:25 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-07T20:53:25.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:25 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-07T20:53:25.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:25 smithi177 ceph-mon[29851]: osdmap e70: 8 total, 8 up, 8 in 2024-04-07T20:53:25.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:25 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-07T20:53:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:25 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-07T20:53:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:25 smithi187 ceph-mon[25907]: osdmap e70: 8 total, 8 up, 8 in 2024-04-07T20:53:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:25 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-07T20:53:26.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-07T20:53:26.370 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:26 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-07T20:53:26.370 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:26 smithi177 ceph-mon[29851]: osdmap e71: 8 total, 8 up, 8 in 2024-04-07T20:53:26.370 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:26 smithi177 ceph-mon[29851]: pgmap v77: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-07T20:53:26.370 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:26 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-07T20:53:26.370 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:26 smithi177 ceph-mon[24852]: osdmap e71: 8 total, 8 up, 8 in 2024-04-07T20:53:26.370 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:26 smithi177 ceph-mon[24852]: pgmap v77: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-07T20:53:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:26 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/478942704' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-07T20:53:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:26 smithi187 ceph-mon[25907]: osdmap e71: 8 total, 8 up, 8 in 2024-04-07T20:53:26.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:26 smithi187 ceph-mon[25907]: pgmap v77: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-07T20:53:28.071 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled rgw.foo update... 2024-04-07T20:53:28.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch apply rgw smpl' 2024-04-07T20:53:28.993 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:29.246 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:28] "GET /metrics HTTP/1.1" 200 196881 "" "Prometheus/2.33.4" 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: pgmap v78: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 375 B/s wr, 1 op/s 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: Saving service rgw.foo spec with placement count:2 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: Saving service rgw.foo spec with placement count:2 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: Deploying daemon rgw.foo.smithi177.pbcdof on smithi177 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-07T20:53:29.248 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: pgmap v78: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 375 B/s wr, 1 op/s 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: Saving service rgw.foo spec with placement count:2 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: Saving service rgw.foo spec with placement count:2 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.250 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: Deploying daemon rgw.foo.smithi177.pbcdof on smithi177 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.251 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:53:29.252 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:53:29.252 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:53:29.252 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:53:29.252 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.252 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: pgmap v78: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 375 B/s wr, 1 op/s 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: Saving service rgw.foo spec with placement count:2 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: Saving service rgw.foo spec with placement count:2 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.pbcdof", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: Deploying daemon rgw.foo.smithi177.pbcdof on smithi177 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:29.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:53:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:53:30.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-07T20:53:30.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[29851]: osdmap e72: 8 total, 8 up, 8 in 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-04-07T20:53:30.360 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:30 smithi177 ceph-mon[24852]: osdmap e72: 8 total, 8 up, 8 in 2024-04-07T20:53:30.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-07T20:53:30.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-04-07T20:53:30.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-04-07T20:53:30.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:30 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-04-07T20:53:30.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:30 smithi187 ceph-mon[25907]: osdmap e72: 8 total, 8 up, 8 in 2024-04-07T20:53:31.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:31 smithi177 ceph-mon[29851]: pgmap v80: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-07T20:53:31.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:31 smithi177 ceph-mon[29851]: osdmap e73: 8 total, 8 up, 8 in 2024-04-07T20:53:31.367 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:31 smithi177 ceph-mon[24852]: pgmap v80: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-07T20:53:31.367 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:31 smithi177 ceph-mon[24852]: osdmap e73: 8 total, 8 up, 8 in 2024-04-07T20:53:31.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:31 smithi187 ceph-mon[25907]: pgmap v80: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-07T20:53:31.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:31 smithi187 ceph-mon[25907]: osdmap e73: 8 total, 8 up, 8 in 2024-04-07T20:53:32.077 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled rgw.smpl update... 2024-04-07T20:53:32.992 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph osd pool create foo' 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: pgmap v82: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='client.24616 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: Saving service rgw.smpl spec with placement count:2 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: pgmap v82: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='client.24616 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: Saving service rgw.smpl spec with placement count:2 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:33 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: pgmap v82: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='client.24616 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: Saving service rgw.smpl spec with placement count:2 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi187.xszzor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:33.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:33 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:34.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:34 smithi177 ceph-mon[29851]: Deploying daemon rgw.foo.smithi187.xszzor on smithi187 2024-04-07T20:53:34.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:34 smithi177 ceph-mon[24852]: Deploying daemon rgw.foo.smithi187.xszzor on smithi187 2024-04-07T20:53:34.431 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:34 smithi187 ceph-mon[25907]: Deploying daemon rgw.foo.smithi187.xszzor on smithi187 2024-04-07T20:53:35.342 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:35 smithi177 ceph-mon[29851]: pgmap v83: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-07T20:53:35.343 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:35 smithi177 ceph-mon[24852]: pgmap v83: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-07T20:53:35.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:35 smithi187 ceph-mon[25907]: pgmap v83: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-07T20:53:36.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: pgmap v84: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 5.0 KiB/s wr, 142 op/s 2024-04-07T20:53:36.266 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.267 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:36 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:36.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:36.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:36.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:36.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.452 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: pgmap v84: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 5.0 KiB/s wr, 142 op/s 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:36.453 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: pgmap v84: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 5.0 KiB/s wr, 142 op/s 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:36.454 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:36 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:37.636 INFO:teuthology.orchestra.run.smithi177.stderr:pool 'foo' created 2024-04-07T20:53:37.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:37 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/114587916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-07T20:53:37.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:37 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-07T20:53:38.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:37 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/114587916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-07T20:53:38.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:37 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-07T20:53:38.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:37 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/114587916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-07T20:53:38.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:37 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-07T20:53:38.438 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'rbd pool init foo' 2024-04-07T20:53:38.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[29851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[29851]: osdmap e74: 8 total, 8 up, 8 in 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[29851]: pgmap v86: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[24852]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[24852]: osdmap e74: 8 total, 8 up, 8 in 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[24852]: pgmap v86: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-04-07T20:53:38.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:38.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:38 smithi187 ceph-mon[25907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-07T20:53:38.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:38 smithi187 ceph-mon[25907]: osdmap e74: 8 total, 8 up, 8 in 2024-04-07T20:53:38.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:38 smithi187 ceph-mon[25907]: pgmap v86: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-04-07T20:53:38.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:38 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:39.020 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:38 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:38] "GET /metrics HTTP/1.1" 200 196881 "" "Prometheus/2.33.4" 2024-04-07T20:53:39.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:39 smithi187 ceph-mon[25907]: osdmap e75: 8 total, 8 up, 8 in 2024-04-07T20:53:39.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:53:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:53:40.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:39 smithi177 ceph-mon[24852]: osdmap e75: 8 total, 8 up, 8 in 2024-04-07T20:53:40.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:39 smithi177 ceph-mon[29851]: osdmap e75: 8 total, 8 up, 8 in 2024-04-07T20:53:40.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:40 smithi177 ceph-mon[29851]: osdmap e76: 8 total, 8 up, 8 in 2024-04-07T20:53:40.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:40 smithi177 ceph-mon[29851]: pgmap v89: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 6.0 KiB/s wr, 366 op/s 2024-04-07T20:53:40.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:40 smithi177 ceph-mon[24852]: osdmap e76: 8 total, 8 up, 8 in 2024-04-07T20:53:40.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:40 smithi177 ceph-mon[24852]: pgmap v89: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 6.0 KiB/s wr, 366 op/s 2024-04-07T20:53:40.766 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:40 smithi187 ceph-mon[25907]: osdmap e76: 8 total, 8 up, 8 in 2024-04-07T20:53:40.767 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:40 smithi187 ceph-mon[25907]: pgmap v89: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 6.0 KiB/s wr, 366 op/s 2024-04-07T20:53:41.437 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:41 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:53:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-mon[24852]: pgmap v90: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3607869796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-mon[29851]: pgmap v90: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-07T20:53:42.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:42 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3607869796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-07T20:53:42.550 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:42 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:42.550 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:42 smithi187 ceph-mon[25907]: pgmap v90: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-07T20:53:42.550 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:42 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3607869796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-07T20:53:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:43 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3607869796' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-07T20:53:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:43 smithi187 ceph-mon[25907]: osdmap e77: 8 total, 8 up, 8 in 2024-04-07T20:53:43.758 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3607869796' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-07T20:53:43.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-mon[24852]: osdmap e77: 8 total, 8 up, 8 in 2024-04-07T20:53:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3607869796' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-07T20:53:43.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-mon[29851]: osdmap e77: 8 total, 8 up, 8 in 2024-04-07T20:53:44.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:44.759 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:53:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:45.112 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: pgmap v92: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-07T20:53:45.112 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.112 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.112 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: osdmap e78: 8 total, 8 up, 8 in 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:45.113 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: pgmap v92: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: osdmap e78: 8 total, 8 up, 8 in 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:45.114 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.115 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:44 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:45.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: pgmap v92: 161 pgs: 10 creating+activating, 151 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: osdmap e78: 8 total, 8 up, 8 in 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi177.soabjy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:45.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:45.308 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:44 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:46.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-07T20:53:46.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:46 smithi177 ceph-mon[29851]: Saving service rgw.smpl spec with placement count:2 2024-04-07T20:53:46.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:46 smithi177 ceph-mon[29851]: Deploying daemon rgw.smpl.smithi177.soabjy on smithi177 2024-04-07T20:53:46.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:46 smithi177 ceph-mon[29851]: osdmap e79: 8 total, 8 up, 8 in 2024-04-07T20:53:46.180 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:46 smithi177 ceph-mon[24852]: Saving service rgw.smpl spec with placement count:2 2024-04-07T20:53:46.180 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:46 smithi177 ceph-mon[24852]: Deploying daemon rgw.smpl.smithi177.soabjy on smithi177 2024-04-07T20:53:46.180 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:46 smithi177 ceph-mon[24852]: osdmap e79: 8 total, 8 up, 8 in 2024-04-07T20:53:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:46 smithi187 ceph-mon[25907]: Saving service rgw.smpl spec with placement count:2 2024-04-07T20:53:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:46 smithi187 ceph-mon[25907]: Deploying daemon rgw.smpl.smithi177.soabjy on smithi177 2024-04-07T20:53:46.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:46 smithi187 ceph-mon[25907]: osdmap e79: 8 total, 8 up, 8 in 2024-04-07T20:53:47.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:47 smithi177 ceph-mon[29851]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:47.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:47 smithi177 ceph-mon[24852]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:47.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:47 smithi187 ceph-mon[25907]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:48.623 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled iscsi.foo update... 2024-04-07T20:53:49.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:48] "GET /metrics HTTP/1.1" 200 208165 "" "Prometheus/2.33.4" 2024-04-07T20:53:49.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:49 smithi177 ceph-mon[24852]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:49.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:49 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:49.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:49 smithi177 ceph-mon[29851]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:49.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:49 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:49.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'sleep 120' 2024-04-07T20:53:49.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:49 smithi187 ceph-mon[25907]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:53:49.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:49 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:49.752 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:53:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: Saving service iscsi.foo spec with placement count:1 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:50.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: Saving service iscsi.foo spec with placement count:1 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:50.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:50 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:50.485 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:53:50.485 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: Saving service iscsi.foo spec with placement count:1 2024-04-07T20:53:50.485 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:50.485 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:50.485 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T20:53:50.486 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi187.fluayj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T20:53:50.486 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:50.486 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:50 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:51.188 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:51 smithi187 ceph-mon[25907]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 313 B/s wr, 0 op/s 2024-04-07T20:53:51.189 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:51 smithi187 ceph-mon[25907]: Deploying daemon rgw.smpl.smithi187.fluayj on smithi187 2024-04-07T20:53:51.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:51 smithi177 ceph-mon[29851]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 313 B/s wr, 0 op/s 2024-04-07T20:53:51.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:51 smithi177 ceph-mon[29851]: Deploying daemon rgw.smpl.smithi187.fluayj on smithi187 2024-04-07T20:53:51.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:51 smithi177 ceph-mon[24852]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 313 B/s wr, 0 op/s 2024-04-07T20:53:51.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:51 smithi177 ceph-mon[24852]: Deploying daemon rgw.smpl.smithi187.fluayj on smithi187 2024-04-07T20:53:53.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-07T20:53:53.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: osdmap e80: 8 total, 8 up, 8 in 2024-04-07T20:53:53.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:53.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:53.272 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.272 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:53.272 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:53.272 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.414 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-07T20:53:53.414 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: osdmap e80: 8 total, 8 up, 8 in 2024-04-07T20:53:53.414 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.414 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:53.414 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:53.414 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: osdmap e80: 8 total, 8 up, 8 in 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:53.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:54.247 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:54 smithi187 ceph-mon[25907]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:54.247 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:54 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:54.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:54 smithi187 ceph-mon[25907]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:54.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:54 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:54.248 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:54 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:54.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[24852]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:54.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[24852]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[29851]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[29851]: Checking dashboard <-> RGW credentials 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:53:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:54 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:55.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:55 smithi187 ceph-mon[25907]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 239 B/s wr, 0 op/s 2024-04-07T20:53:55.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:55 smithi177 ceph-mon[29851]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 239 B/s wr, 0 op/s 2024-04-07T20:53:55.400 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:55 smithi177 ceph-mon[24852]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 239 B/s wr, 0 op/s 2024-04-07T20:53:56.173 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:55 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:53:55] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:56.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:56] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:57.254 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:56 smithi187 ceph-mon[25907]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 409 B/s wr, 208 op/s 2024-04-07T20:53:57.254 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:56 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:57.254 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:56 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:57.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-mon[24852]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 409 B/s wr, 208 op/s 2024-04-07T20:53:57.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:57.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:57.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-mon[29851]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 409 B/s wr, 208 op/s 2024-04-07T20:53:57.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:57.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:56 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:58.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:57 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:53:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:58.948 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:53:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:53:59.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:58] "GET /metrics HTTP/1.1" 200 215034 "" "Prometheus/2.33.4" 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 409 B/s wr, 208 op/s 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-07T20:53:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 409 B/s wr, 208 op/s 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-07T20:53:59.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:58 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:59.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 409 B/s wr, 208 op/s 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi177.kntdfp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-07T20:53:59.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:58 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:53:59.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:53:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:53:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:54:00.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:59 smithi177 ceph-mon[24852]: Checking pool "foo" exists for service iscsi.foo 2024-04-07T20:54:00.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:53:59 smithi177 ceph-mon[24852]: Deploying daemon iscsi.foo.smithi177.kntdfp on smithi177 2024-04-07T20:54:00.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:59 smithi177 ceph-mon[29851]: Checking pool "foo" exists for service iscsi.foo 2024-04-07T20:54:00.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:53:59 smithi177 ceph-mon[29851]: Deploying daemon iscsi.foo.smithi177.kntdfp on smithi177 2024-04-07T20:54:00.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:59 smithi187 ceph-mon[25907]: Checking pool "foo" exists for service iscsi.foo 2024-04-07T20:54:00.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:53:59 smithi187 ceph-mon[25907]: Deploying daemon iscsi.foo.smithi177.kntdfp on smithi177 2024-04-07T20:54:01.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:00 smithi177 ceph-mon[24852]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:01.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:00 smithi177 ceph-mon[29851]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:01.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:00 smithi187 ceph-mon[25907]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:02.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:01 smithi177 ceph-mon[24852]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-07T20:54:02.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:01 smithi177 ceph-mon[29851]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-07T20:54:02.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:01 smithi187 ceph-mon[25907]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-07T20:54:03.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:03.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:03.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:03.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:54:03.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: Adding iSCSI gateway http://:@172.21.15.177:5000 to Dashboard 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi177"}]: dispatch 2024-04-07T20:54:03.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi177"}]: dispatch 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3548229077' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2566213391' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]: dispatch 2024-04-07T20:54:03.512 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]: dispatch 2024-04-07T20:54:03.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:03.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:03.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:03.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: Adding iSCSI gateway http://:@172.21.15.177:5000 to Dashboard 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-07T20:54:03.514 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi177"}]: dispatch 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi177"}]: dispatch 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:54:03.515 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3548229077' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-07T20:54:03.516 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2566213391' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]: dispatch 2024-04-07T20:54:03.516 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:03 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]: dispatch 2024-04-07T20:54:03.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: Adding iSCSI gateway http://:@172.21.15.177:5000 to Dashboard 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi177"}]: dispatch 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi177"}]: dispatch 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3548229077' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2566213391' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]: dispatch 2024-04-07T20:54:03.558 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:03 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]: dispatch 2024-04-07T20:54:04.243 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]': finished 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[29851]: osdmap e81: 8 total, 8 up, 8 in 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3873400303' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3514106323"}]: dispatch 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]': finished 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[24852]: osdmap e81: 8 total, 8 up, 8 in 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:04.244 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:04 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3873400303' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3514106323"}]: dispatch 2024-04-07T20:54:04.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:04 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1769461466"}]': finished 2024-04-07T20:54:04.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:04 smithi187 ceph-mon[25907]: osdmap e81: 8 total, 8 up, 8 in 2024-04-07T20:54:04.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:04.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:04 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3873400303' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3514106323"}]: dispatch 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[29851]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3873400303' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3514106323"}]': finished 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[29851]: osdmap e82: 8 total, 8 up, 8 in 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1526336674' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1236583948"}]: dispatch 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[24852]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3873400303' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3514106323"}]': finished 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[24852]: osdmap e82: 8 total, 8 up, 8 in 2024-04-07T20:54:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1526336674' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1236583948"}]: dispatch 2024-04-07T20:54:05.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:05 smithi187 ceph-mon[25907]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-07T20:54:05.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:05 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3873400303' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3514106323"}]': finished 2024-04-07T20:54:05.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:05 smithi187 ceph-mon[25907]: osdmap e82: 8 total, 8 up, 8 in 2024-04-07T20:54:05.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:05 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1526336674' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1236583948"}]: dispatch 2024-04-07T20:54:06.008 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:05 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:54:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:06.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1526336674' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1236583948"}]': finished 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[24852]: osdmap e83: 8 total, 8 up, 8 in 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3086308864' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/851605445"}]: dispatch 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[24852]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1526336674' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1236583948"}]': finished 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[29851]: osdmap e83: 8 total, 8 up, 8 in 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3086308864' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/851605445"}]: dispatch 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:06.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:06 smithi177 ceph-mon[29851]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-04-07T20:54:06.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:06 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1526336674' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1236583948"}]': finished 2024-04-07T20:54:06.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:06 smithi187 ceph-mon[25907]: osdmap e83: 8 total, 8 up, 8 in 2024-04-07T20:54:06.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:06 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3086308864' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/851605445"}]: dispatch 2024-04-07T20:54:06.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:06 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:06.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:06 smithi187 ceph-mon[25907]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-04-07T20:54:07.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:07 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3086308864' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/851605445"}]': finished 2024-04-07T20:54:07.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:07 smithi177 ceph-mon[29851]: osdmap e84: 8 total, 8 up, 8 in 2024-04-07T20:54:07.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:07 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2505970260' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1872289908"}]: dispatch 2024-04-07T20:54:07.296 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:07 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3086308864' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/851605445"}]': finished 2024-04-07T20:54:07.296 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:07 smithi177 ceph-mon[24852]: osdmap e84: 8 total, 8 up, 8 in 2024-04-07T20:54:07.297 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:07 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2505970260' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1872289908"}]: dispatch 2024-04-07T20:54:07.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:07 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3086308864' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/851605445"}]': finished 2024-04-07T20:54:07.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:07 smithi187 ceph-mon[25907]: osdmap e84: 8 total, 8 up, 8 in 2024-04-07T20:54:07.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:07 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2505970260' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1872289908"}]: dispatch 2024-04-07T20:54:08.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:54:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2505970260' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1872289908"}]': finished 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[29851]: osdmap e85: 8 total, 8 up, 8 in 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2942722319' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]: dispatch 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]: dispatch 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[29851]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 2 op/s 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2505970260' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1872289908"}]': finished 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[24852]: osdmap e85: 8 total, 8 up, 8 in 2024-04-07T20:54:08.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2942722319' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]: dispatch 2024-04-07T20:54:08.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]: dispatch 2024-04-07T20:54:08.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-mon[24852]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 2 op/s 2024-04-07T20:54:08.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:08 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2505970260' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1872289908"}]': finished 2024-04-07T20:54:08.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:08 smithi187 ceph-mon[25907]: osdmap e85: 8 total, 8 up, 8 in 2024-04-07T20:54:08.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:08 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2942722319' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]: dispatch 2024-04-07T20:54:08.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:08 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]: dispatch 2024-04-07T20:54:08.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:08 smithi187 ceph-mon[25907]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 255 B/s wr, 2 op/s 2024-04-07T20:54:09.254 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:08] "GET /metrics HTTP/1.1" 200 215034 "" "Prometheus/2.33.4" 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]': finished 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: osdmap e86: 8 total, 8 up, 8 in 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2875909016' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]: dispatch 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]: dispatch 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]': finished 2024-04-07T20:54:09.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: osdmap e86: 8 total, 8 up, 8 in 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2875909016' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]: dispatch 2024-04-07T20:54:09.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:09 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]: dispatch 2024-04-07T20:54:09.556 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:54:09.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3006842145"}]': finished 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: osdmap e86: 8 total, 8 up, 8 in 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2875909016' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]: dispatch 2024-04-07T20:54:09.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:09 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]: dispatch 2024-04-07T20:54:10.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]': finished 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[29851]: osdmap e87: 8 total, 8 up, 8 in 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[29851]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2495533864' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1166756411"}]: dispatch 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]': finished 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[24852]: osdmap e87: 8 total, 8 up, 8 in 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[24852]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:10.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:10 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2495533864' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1166756411"}]: dispatch 2024-04-07T20:54:10.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:10 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/2181256441"}]': finished 2024-04-07T20:54:10.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:10 smithi187 ceph-mon[25907]: osdmap e87: 8 total, 8 up, 8 in 2024-04-07T20:54:10.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:10 smithi187 ceph-mon[25907]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:10.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:10 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2495533864' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1166756411"}]: dispatch 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2495533864' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1166756411"}]': finished 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[24852]: osdmap e88: 8 total, 8 up, 8 in 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/256668403' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]: dispatch 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]: dispatch 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2495533864' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1166756411"}]': finished 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[29851]: osdmap e88: 8 total, 8 up, 8 in 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/256668403' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]: dispatch 2024-04-07T20:54:11.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:11 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]: dispatch 2024-04-07T20:54:11.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:11 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2495533864' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/1166756411"}]': finished 2024-04-07T20:54:11.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:11 smithi187 ceph-mon[25907]: osdmap e88: 8 total, 8 up, 8 in 2024-04-07T20:54:11.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:11 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/256668403' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]: dispatch 2024-04-07T20:54:11.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:11 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]: dispatch 2024-04-07T20:54:12.423 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:54:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]': finished 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[24852]: osdmap e89: 8 total, 8 up, 8 in 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[24852]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/595087101' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/646064095"}]: dispatch 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]': finished 2024-04-07T20:54:12.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[29851]: osdmap e89: 8 total, 8 up, 8 in 2024-04-07T20:54:12.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[29851]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:12.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/595087101' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/646064095"}]: dispatch 2024-04-07T20:54:12.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:12.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:12 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:12.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:12 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3006842145"}]': finished 2024-04-07T20:54:12.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:12 smithi187 ceph-mon[25907]: osdmap e89: 8 total, 8 up, 8 in 2024-04-07T20:54:12.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:12 smithi187 ceph-mon[25907]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:12.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:12 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/595087101' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/646064095"}]: dispatch 2024-04-07T20:54:12.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:12 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:12.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:12 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:13.496 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:13 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:13.497 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:13 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/595087101' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/646064095"}]': finished 2024-04-07T20:54:13.497 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:13 smithi187 ceph-mon[25907]: osdmap e90: 8 total, 8 up, 8 in 2024-04-07T20:54:13.497 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:13 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/905827546' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3203112426"}]: dispatch 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/595087101' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/646064095"}]': finished 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[29851]: osdmap e90: 8 total, 8 up, 8 in 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/905827546' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3203112426"}]: dispatch 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/595087101' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/646064095"}]': finished 2024-04-07T20:54:13.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[24852]: osdmap e90: 8 total, 8 up, 8 in 2024-04-07T20:54:13.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:13 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/905827546' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3203112426"}]: dispatch 2024-04-07T20:54:14.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:14 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/905827546' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3203112426"}]': finished 2024-04-07T20:54:14.657 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:14 smithi187 ceph-mon[25907]: osdmap e91: 8 total, 8 up, 8 in 2024-04-07T20:54:14.657 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:14 smithi187 ceph-mon[25907]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:54:14.657 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:14 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3273286498' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]: dispatch 2024-04-07T20:54:14.657 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:14 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]: dispatch 2024-04-07T20:54:14.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/905827546' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3203112426"}]': finished 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[29851]: osdmap e91: 8 total, 8 up, 8 in 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[29851]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3273286498' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]: dispatch 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]: dispatch 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/905827546' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3203112426"}]': finished 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[24852]: osdmap e91: 8 total, 8 up, 8 in 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[24852]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3273286498' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]: dispatch 2024-04-07T20:54:14.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:14 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]: dispatch 2024-04-07T20:54:15.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:15 smithi177 ceph-mon[24852]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]': finished 2024-04-07T20:54:15.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:15 smithi177 ceph-mon[24852]: osdmap e92: 8 total, 8 up, 8 in 2024-04-07T20:54:15.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:15 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1140696287' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3172484468"}]: dispatch 2024-04-07T20:54:15.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:15 smithi177 ceph-mon[29851]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]': finished 2024-04-07T20:54:15.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:15 smithi177 ceph-mon[29851]: osdmap e92: 8 total, 8 up, 8 in 2024-04-07T20:54:15.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:15 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1140696287' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3172484468"}]: dispatch 2024-04-07T20:54:15.775 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:15 smithi187 ceph-mon[25907]: from='client.? ' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/2181256441"}]': finished 2024-04-07T20:54:15.776 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:15 smithi187 ceph-mon[25907]: osdmap e92: 8 total, 8 up, 8 in 2024-04-07T20:54:15.776 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:15 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1140696287' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3172484468"}]: dispatch 2024-04-07T20:54:16.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1140696287' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3172484468"}]': finished 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[29851]: osdmap e93: 8 total, 8 up, 8 in 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[29851]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2907223272' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3045474916"}]: dispatch 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1140696287' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3172484468"}]': finished 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[24852]: osdmap e93: 8 total, 8 up, 8 in 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[24852]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:16.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:16 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2907223272' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3045474916"}]: dispatch 2024-04-07T20:54:16.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:16 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1140696287' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/3172484468"}]': finished 2024-04-07T20:54:16.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:16 smithi187 ceph-mon[25907]: osdmap e93: 8 total, 8 up, 8 in 2024-04-07T20:54:16.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:16 smithi187 ceph-mon[25907]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:16.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:16 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2907223272' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3045474916"}]: dispatch 2024-04-07T20:54:17.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:17 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2907223272' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3045474916"}]': finished 2024-04-07T20:54:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:17 smithi177 ceph-mon[29851]: osdmap e94: 8 total, 8 up, 8 in 2024-04-07T20:54:17.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:17 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/4079449014' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/502485757"}]: dispatch 2024-04-07T20:54:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:17 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2907223272' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3045474916"}]': finished 2024-04-07T20:54:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:17 smithi177 ceph-mon[24852]: osdmap e94: 8 total, 8 up, 8 in 2024-04-07T20:54:17.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:17 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4079449014' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/502485757"}]: dispatch 2024-04-07T20:54:17.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:17 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2907223272' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6800/3045474916"}]': finished 2024-04-07T20:54:17.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:17 smithi187 ceph-mon[25907]: osdmap e94: 8 total, 8 up, 8 in 2024-04-07T20:54:17.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:17 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/4079449014' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/502485757"}]: dispatch 2024-04-07T20:54:18.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/4079449014' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/502485757"}]': finished 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[29851]: osdmap e95: 8 total, 8 up, 8 in 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[29851]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2032343795' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3045474916"}]: dispatch 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4079449014' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/502485757"}]': finished 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[24852]: osdmap e95: 8 total, 8 up, 8 in 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[24852]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:18.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2032343795' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3045474916"}]: dispatch 2024-04-07T20:54:18.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:18 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/4079449014' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:0/502485757"}]': finished 2024-04-07T20:54:18.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:18 smithi187 ceph-mon[25907]: osdmap e95: 8 total, 8 up, 8 in 2024-04-07T20:54:18.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:18 smithi187 ceph-mon[25907]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:18.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:18 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2032343795' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3045474916"}]: dispatch 2024-04-07T20:54:19.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:18] "GET /metrics HTTP/1.1" 200 215059 "" "Prometheus/2.33.4" 2024-04-07T20:54:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:19 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2032343795' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3045474916"}]': finished 2024-04-07T20:54:19.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:19 smithi177 ceph-mon[29851]: osdmap e96: 8 total, 8 up, 8 in 2024-04-07T20:54:19.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:19 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2032343795' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3045474916"}]': finished 2024-04-07T20:54:19.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:19 smithi177 ceph-mon[24852]: osdmap e96: 8 total, 8 up, 8 in 2024-04-07T20:54:19.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:19 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2032343795' entity='client.iscsi.foo.smithi177.kntdfp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.177:6801/3045474916"}]': finished 2024-04-07T20:54:19.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:19 smithi187 ceph-mon[25907]: osdmap e96: 8 total, 8 up, 8 in 2024-04-07T20:54:19.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:54:19 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:54:20.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:20 smithi177 ceph-mon[29851]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:20.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:20 smithi177 ceph-mon[24852]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:20.807 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:20 smithi187 ceph-mon[25907]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:22.893 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:54:22] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-mon[24852]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-mon[29851]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:23.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:23 smithi187 ceph-mon[25907]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:23.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:23 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:23.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:23 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:23.868 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:23.868 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:54:24.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:24 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:24.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:24 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:24 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:25 smithi177 ceph-mon[24852]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 0 op/s 2024-04-07T20:54:25.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:25 smithi177 ceph-mon[29851]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 0 op/s 2024-04-07T20:54:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:25 smithi187 ceph-mon[25907]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 0 op/s 2024-04-07T20:54:27.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:27 smithi177 ceph-mon[24852]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:27.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:27 smithi177 ceph-mon[29851]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:27.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:27 smithi187 ceph-mon[25907]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:29.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:28] "GET /metrics HTTP/1.1" 200 215048 "" "Prometheus/2.33.4" 2024-04-07T20:54:29.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[29851]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[24852]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:54:29.260 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:54:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:29 smithi187 ceph-mon[25907]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-07T20:54:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:54:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:54:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:54:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:54:29.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:54:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:54:31.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:31 smithi177 ceph-mon[24852]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-07T20:54:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:31 smithi177 ceph-mon[29851]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-07T20:54:31.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:31 smithi187 ceph-mon[25907]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-07T20:54:32.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:54:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:33.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:33 smithi187 ceph-mon[25907]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:33.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:33.463 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:33.506 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-mon[29851]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:33.506 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:33.506 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:33.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-mon[24852]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:33.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:33.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:33.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:54:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:33.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:33.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:54:34.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:34 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:34.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:34 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:34.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:34 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:35.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:35 smithi177 ceph-mon[24852]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:35.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:35 smithi177 ceph-mon[29851]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:35.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:35 smithi187 ceph-mon[25907]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:37.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:37 smithi177 ceph-mon[24852]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:37.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:37 smithi177 ceph-mon[29851]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:37.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:37 smithi187 ceph-mon[25907]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:39.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:39 smithi177 ceph-mon[24852]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:39.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:39 smithi177 ceph-mon[29851]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:39.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:38 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:38] "GET /metrics HTTP/1.1" 200 215048 "" "Prometheus/2.33.4" 2024-04-07T20:54:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:39 smithi187 ceph-mon[25907]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:39.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:54:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:54:41.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:41 smithi177 ceph-mon[24852]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:41.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:41 smithi177 ceph-mon[29851]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:41.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:41 smithi187 ceph-mon[25907]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:42.932 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:42 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:54:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:43.506 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-mon[29851]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:43.507 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:43.507 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:43.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-mon[24852]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:43.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:43.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:43 smithi187 ceph-mon[25907]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:43.758 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:54:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:43.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:43.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:54:44.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:44 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:44.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:44 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:44 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:45.476 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:45 smithi177 ceph-mon[24852]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:45.476 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:45 smithi177 ceph-mon[29851]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:45 smithi187 ceph-mon[25907]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:46.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:46 smithi177 ceph-mon[24852]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:46.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:46 smithi177 ceph-mon[29851]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:46.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:46 smithi187 ceph-mon[25907]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:49.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:49 smithi177 ceph-mon[24852]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:49.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:49 smithi177 ceph-mon[29851]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:49.260 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:48] "GET /metrics HTTP/1.1" 200 215007 "" "Prometheus/2.33.4" 2024-04-07T20:54:49.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:49 smithi187 ceph-mon[25907]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:49.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:54:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:54:51.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:51 smithi177 ceph-mon[29851]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:51.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:51 smithi177 ceph-mon[24852]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:51 smithi187 ceph-mon[25907]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:52.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:54:52] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:54:53.405 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:53 smithi187 ceph-mon[25907]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:53.405 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:53.405 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:53.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-mon[24852]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:53.507 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:53.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:53.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-mon[29851]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:53.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:53.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:54:53.758 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:54:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:53.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:54:53.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:54:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:54:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:54:54.416 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:54 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:54.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:54 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:54.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:54 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:54:55.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:55 smithi177 ceph-mon[29851]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:55.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:55 smithi177 ceph-mon[24852]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:55 smithi187 ceph-mon[25907]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:57.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:57 smithi177 ceph-mon[29851]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:57.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:57 smithi177 ceph-mon[24852]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:57.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:57 smithi187 ceph-mon[25907]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:54:59.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:54:59 smithi177 ceph-mon[24852]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:54:59 smithi177 ceph-mon[29851]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:59.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:54:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:58] "GET /metrics HTTP/1.1" 200 214956 "" "Prometheus/2.33.4" 2024-04-07T20:54:59.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:54:59 smithi187 ceph-mon[25907]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:54:59.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:54:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:54:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:55:01.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:01 smithi177 ceph-mon[24852]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:01.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:01 smithi177 ceph-mon[29851]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:01.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:01 smithi187 ceph-mon[25907]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:02.886 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:02 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:55:02] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:55:03.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-mon[24852]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:03.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:03.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:03.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-mon[29851]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:03.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:03.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:03.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:03 smithi187 ceph-mon[25907]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:03.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:03 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:03.827 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:55:03.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:03.827 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:03.827 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:04.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:04 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:04 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:04 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:05.371 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:05 smithi177 ceph-mon[29851]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:05.371 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:05 smithi177 ceph-mon[24852]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:05.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:05 smithi187 ceph-mon[25907]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:07.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:07 smithi187 ceph-mon[25907]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:07.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:07 smithi177 ceph-mon[24852]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:07.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:07 smithi177 ceph-mon[29851]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:09.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:08] "GET /metrics HTTP/1.1" 200 214956 "" "Prometheus/2.33.4" 2024-04-07T20:55:09.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[24852]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:09.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:55:09.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:55:09.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:09.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[29851]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:09.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:55:09.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:55:09.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:09 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:09.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:09 smithi187 ceph-mon[25907]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:09.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:55:09.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:09 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:55:09.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:09 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:09.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:55:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:55:11.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:11 smithi177 ceph-mon[24852]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:11.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:11 smithi177 ceph-mon[29851]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:11.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:11 smithi187 ceph-mon[25907]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:12.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:55:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:13 smithi187 ceph-mon[25907]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:13 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:13 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:13.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-mon[29851]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-mon[24852]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:14.008 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:55:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:14.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:14 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:14.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:14 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:14.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:14 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:15.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:15 smithi177 ceph-mon[24852]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:15.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:15 smithi177 ceph-mon[29851]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:15.561 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:15 smithi187 ceph-mon[25907]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:16.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:16 smithi177 ceph-mon[29851]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:16.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:16 smithi177 ceph-mon[24852]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:16.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:16 smithi187 ceph-mon[25907]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:19.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:19 smithi177 ceph-mon[24852]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:19.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:19 smithi177 ceph-mon[29851]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:19.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:18] "GET /metrics HTTP/1.1" 200 214978 "" "Prometheus/2.33.4" 2024-04-07T20:55:19.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:19 smithi187 ceph-mon[25907]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:19.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:55:19 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:55:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:21 smithi177 ceph-mon[24852]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:21 smithi177 ceph-mon[29851]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:21 smithi187 ceph-mon[25907]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:23.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-mon[24852]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:23.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:23.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:23.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:22 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:55:22] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:55:23.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-mon[29851]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:23.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:23.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:23.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:23 smithi187 ceph-mon[25907]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:23.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:23 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:23.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:23 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:23.758 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:55:23.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:23.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:23.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:24.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:24 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:24.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:24 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:24 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:25.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:25 smithi177 ceph-mon[29851]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:25.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:25 smithi177 ceph-mon[24852]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:25.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:25 smithi187 ceph-mon[25907]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:27.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:27 smithi177 ceph-mon[29851]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:27.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:27 smithi177 ceph-mon[24852]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:27.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:27 smithi187 ceph-mon[25907]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:29.174 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:28] "GET /metrics HTTP/1.1" 200 214997 "" "Prometheus/2.33.4" 2024-04-07T20:55:29.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:29 smithi187 ceph-mon[25907]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:55:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:55:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:55:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:55:29.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[29851]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[24852]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:55:29.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:55:29.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:55:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:55:31.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:31 smithi177 ceph-mon[24852]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:31.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:31 smithi177 ceph-mon[29851]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:31.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:31 smithi187 ceph-mon[25907]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:33.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:55:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:55:33.504 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:33 smithi187 ceph-mon[25907]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:33.504 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:33.504 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:33.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-mon[24852]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:33.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:33.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-mon[29851]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:55:33.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:34.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:34 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:34.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:34 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:34.520 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:34 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:35.461 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:35 smithi187 ceph-mon[25907]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:35.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:35 smithi177 ceph-mon[24852]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:35.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:35 smithi177 ceph-mon[29851]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:37.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:37 smithi177 ceph-mon[24852]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:37.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:37 smithi177 ceph-mon[29851]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:37.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:37 smithi187 ceph-mon[25907]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:39.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:39 smithi177 ceph-mon[24852]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:39.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:38 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:38] "GET /metrics HTTP/1.1" 200 214997 "" "Prometheus/2.33.4" 2024-04-07T20:55:39.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:39 smithi177 ceph-mon[29851]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:39.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:39 smithi187 ceph-mon[25907]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:39.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:55:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:55:41.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:41 smithi177 ceph-mon[24852]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:41.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:41 smithi177 ceph-mon[29851]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:41.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:41 smithi187 ceph-mon[25907]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:43.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:55:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:55:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-mon[24852]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:43.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-mon[29851]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:43.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:43.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:43 smithi187 ceph-mon[25907]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:43 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:55:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:44.501 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:44 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:44.502 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:44 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:44 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:45.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:45 smithi177 ceph-mon[29851]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:45.433 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:45 smithi177 ceph-mon[24852]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:45 smithi187 ceph-mon[25907]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:47.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:47 smithi187 ceph-mon[25907]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:47.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:47 smithi177 ceph-mon[24852]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:47.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:47 smithi177 ceph-mon[29851]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:49.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:49 smithi177 ceph-mon[24852]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:49.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:49 smithi177 ceph-mon[29851]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:49.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:48] "GET /metrics HTTP/1.1" 200 215002 "" "Prometheus/2.33.4" 2024-04-07T20:55:49.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:49 smithi187 ceph-mon[25907]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:49.809 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:55:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:55:51.426 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:51 smithi177 ceph-mon[29851]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:51.426 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:51 smithi177 ceph-mon[24852]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:51 smithi187 ceph-mon[25907]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:51.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-07T20:55:52.758 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:55:52] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:55:53.491 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-mon[24852]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:53.492 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:53.492 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:53.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-mon[29851]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:53.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:53.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:53.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:53 smithi187 ceph-mon[25907]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:53.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:53.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:55:53.758 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:55:53.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:53.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:55:53.759 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:55:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:55:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:55:54.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:54 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:54.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:54 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:54.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:54 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:55:54.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-07T20:55:55.470 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:55 smithi187 ceph-mon[25907]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:55.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:55 smithi177 ceph-mon[24852]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:55.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:55 smithi177 ceph-mon[29851]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:57.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-07T20:55:57.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:57 smithi177 ceph-mon[29851]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:57.398 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:57 smithi177 ceph-mon[24852]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:57.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:57 smithi187 ceph-mon[25907]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:55:59.195 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:55:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:58] "GET /metrics HTTP/1.1" 200 214993 "" "Prometheus/2.33.4" 2024-04-07T20:55:59.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:55:59 smithi187 ceph-mon[25907]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:59.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:55:59 smithi177 ceph-mon[24852]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:59.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:55:59 smithi177 ceph-mon[29851]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:55:59.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-07T20:55:59.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:55:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:55:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:56:01.298 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:01 smithi177 ceph-mon[29851]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:01.298 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:01 smithi177 ceph-mon[24852]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:01.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:01 smithi187 ceph-mon[25907]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:01.724 INFO:teuthology.orchestra.run.smithi177.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:02.397 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-07T20:56:02.407 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi177.front.sepia.ceph.com 2024-04-07T20:56:02.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:03.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[29851]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:03.011 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:02 smithi177 ceph-mon[24852]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:03.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:03.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:02 smithi187 ceph-mon[25907]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:56:03.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:56:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:56:03.905 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:56:03.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:03.905 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:03.905 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:04.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:04 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:04 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:04 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:04.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:04.684 INFO:teuthology.orchestra.run.smithi177.stdout:true 2024-04-07T20:56:05.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:05 smithi177 ceph-mon[29851]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:05.440 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:05 smithi177 ceph-mon[24852]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:05.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:05 smithi187 ceph-mon[25907]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:agent.smithi177 smithi177 running 2s ago 9m - - 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:agent.smithi187 smithi187 running 12s ago 8m - - 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:alertmanager.a smithi177 *:9093,9094 running (3m) 2s ago 3m 26.5M - ba2b418f427c bdca865a7e70 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:grafana.a smithi187 *:3000 running (3m) 12s ago 3m 53.9M - 8.3.5 dad864ee21e9 7b41716cc5e1 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:iscsi.foo.smithi177.kntdfp smithi177 running (2m) 2s ago 2m 52.5M - 3.5 e1d6a67b021e d2541e320aef 2024-04-07T20:56:05.575 INFO:teuthology.orchestra.run.smithi177.stdout:mgr.x smithi187 *:8443 running (8m) 12s ago 8m 412M - 17.2.0 e1d6a67b021e a71bc4bae1d4 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:mgr.y smithi177 *:9283 running (9m) 2s ago 9m 464M - 17.2.0 e1d6a67b021e f850f4734a67 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:mon.a smithi177 running (9m) 2s ago 10m 45.4M 2048M 17.2.0 e1d6a67b021e f4208a5ace0f 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:mon.b smithi187 running (8m) 12s ago 8m 41.4M 2048M 17.2.0 e1d6a67b021e a091fb21a3a7 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:mon.c smithi177 running (8m) 2s ago 8m 40.6M 2048M 17.2.0 e1d6a67b021e 5c620a76fdbb 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:node-exporter.a smithi177 *:9100 running (4m) 2s ago 4m 17.0M - 1dbe0e931976 001a28e563be 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:node-exporter.b smithi187 *:9100 running (4m) 12s ago 4m 15.1M - 1dbe0e931976 908396a184b0 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.0 smithi177 running (7m) 2s ago 7m 50.8M 4096M 17.2.0 e1d6a67b021e 82e517b977ab 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.1 smithi177 running (7m) 2s ago 7m 50.9M 4096M 17.2.0 e1d6a67b021e 083118d3b511 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.2 smithi177 running (7m) 2s ago 7m 46.7M 4096M 17.2.0 e1d6a67b021e 8bd183701842 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.3 smithi177 running (6m) 2s ago 6m 49.0M 4096M 17.2.0 e1d6a67b021e b782d950f67a 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.4 smithi187 running (6m) 12s ago 6m 49.9M 4096M 17.2.0 e1d6a67b021e c5615f8077c7 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.5 smithi187 running (5m) 12s ago 5m 48.2M 4096M 17.2.0 e1d6a67b021e fca44dee1afa 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.6 smithi187 running (5m) 12s ago 5m 45.5M 4096M 17.2.0 e1d6a67b021e b45cbc92f590 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:osd.7 smithi187 running (5m) 12s ago 5m 48.9M 4096M 17.2.0 e1d6a67b021e bf0628d56383 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:prometheus.a smithi187 *:9095 running (2m) 12s ago 4m 67.7M - 514e6a882f6e ac118aa18f67 2024-04-07T20:56:05.576 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.foo.smithi177.pbcdof smithi177 *:8000 running (2m) 2s ago 2m 68.2M - 17.2.0 e1d6a67b021e 7f8b02c6bd74 2024-04-07T20:56:05.577 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.foo.smithi187.xszzor smithi187 *:8000 running (2m) 12s ago 2m 70.5M - 17.2.0 e1d6a67b021e a43cdb1e8d0b 2024-04-07T20:56:05.577 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.smpl.smithi177.soabjy smithi177 *:80 running (2m) 2s ago 2m 71.7M - 17.2.0 e1d6a67b021e 8fb2c1b4e46f 2024-04-07T20:56:05.577 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.smpl.smithi187.fluayj smithi187 *:80 running (2m) 12s ago 2m 69.4M - 17.2.0 e1d6a67b021e 3e464562784d 2024-04-07T20:56:06.086 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:56:06.086 INFO:teuthology.orchestra.run.smithi177.stdout: "mon": { 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "mgr": { 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": { 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": {}, 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "rgw": { 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "overall": { 2024-04-07T20:56:06.087 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-07T20:56:06.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:56:06.088 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:56:06.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:06 smithi177 ceph-mon[29851]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:06.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:06 smithi177 ceph-mon[29851]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:06.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:06 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/4249157058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:56:06.493 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:06 smithi177 ceph-mon[24852]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:06.493 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:06 smithi177 ceph-mon[24852]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:06.493 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:06 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/4249157058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:56:06.529 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:56:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", 2024-04-07T20:56:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "in_progress": true, 2024-04-07T20:56:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "services_complete": [], 2024-04-07T20:56:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "progress": "", 2024-04-07T20:56:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d image" 2024-04-07T20:56:06.530 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:56:06.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:06 smithi187 ceph-mon[25907]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:06.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:06 smithi187 ceph-mon[25907]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:06.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:06 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/4249157058' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:56:06.953 INFO:teuthology.orchestra.run.smithi177.stdout:HEALTH_OK 2024-04-07T20:56:07.457 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:07 smithi187 ceph-mon[25907]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:07.457 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:07 smithi187 ceph-mon[25907]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:07.457 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:07 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3201389372' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:56:07.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:07 smithi177 ceph-mon[24852]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:07.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:07 smithi177 ceph-mon[24852]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:07.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:07 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3201389372' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:56:07.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:07 smithi177 ceph-mon[29851]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:07.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:07 smithi177 ceph-mon[29851]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:07.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:07 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3201389372' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:56:08.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:08 smithi177 ceph-mon[24852]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:08.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:08 smithi177 ceph-mon[29851]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:08.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:08 smithi187 ceph-mon[25907]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:09.202 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:08] "GET /metrics HTTP/1.1" 200 214993 "" "Prometheus/2.33.4" 2024-04-07T20:56:09.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:09 smithi187 ceph-mon[25907]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:09.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:09 smithi177 ceph-mon[29851]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:09.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:09 smithi177 ceph-mon[24852]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:09.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:56:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:56:10.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:10 smithi177 ceph-mon[29851]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:10.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:10 smithi177 ceph-mon[24852]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:10.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:10 smithi187 ceph-mon[25907]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:13.172 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:56:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:56:13.482 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:13 smithi187 ceph-mon[25907]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:13.483 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:13 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:13.483 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:13 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-mon[29851]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:13.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-mon[24852]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:13.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:14.008 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:56:13.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:13.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:13.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:14.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:14 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:14.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:14 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:14.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:14 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:15.370 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:15 smithi187 ceph-mon[25907]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:15.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:15 smithi177 ceph-mon[24852]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:15.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:15 smithi177 ceph-mon[29851]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:17.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:17 smithi177 ceph-mon[29851]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:17.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:17 smithi177 ceph-mon[24852]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:17.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:17 smithi187 ceph-mon[25907]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:19.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:19 smithi177 ceph-mon[29851]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:19.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:18] "GET /metrics HTTP/1.1" 200 214979 "" "Prometheus/2.33.4" 2024-04-07T20:56:19.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:19 smithi177 ceph-mon[24852]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:19.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:19 smithi187 ceph-mon[25907]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:19.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:56:19 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:56:21.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:21 smithi177 ceph-mon[29851]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:21.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:21 smithi177 ceph-mon[24852]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:21.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:21 smithi187 ceph-mon[25907]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:23.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:23 smithi177 ceph-mon[29851]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:23.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:56:23] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:56:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:23 smithi177 ceph-mon[24852]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:23.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:23 smithi187 ceph-mon[25907]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:24.008 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:56:23.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:24.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:24.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:24.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:24 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:24.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:24 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:24.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:24 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:24 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:24 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:24.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:24 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:24 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:24 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:24.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:24 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:25.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:25 smithi177 ceph-mon[24852]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:25.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:25 smithi177 ceph-mon[29851]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:25.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:25 smithi187 ceph-mon[25907]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:27.499 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:27 smithi187 ceph-mon[25907]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:27.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:27 smithi177 ceph-mon[29851]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:27.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:27 smithi177 ceph-mon[24852]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:29.245 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:28] "GET /metrics HTTP/1.1" 200 214928 "" "Prometheus/2.33.4" 2024-04-07T20:56:29.246 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[24852]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:29.246 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:56:29.246 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:56:29.246 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:56:29.246 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:56:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[29851]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:56:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:56:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:56:29.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:29 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:56:29.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:29 smithi187 ceph-mon[25907]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:56:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-07T20:56:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:29 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:56:29.460 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:29 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-07T20:56:29.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:56:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:56:31.446 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:31 smithi177 ceph-mon[29851]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:31.447 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:31 smithi177 ceph-mon[24852]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:31.537 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:31 smithi187 ceph-mon[25907]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:32.483 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:32.484 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: Upgrade: Target is version 19.0.0-2578-ge5c885fe (unknown) 2024-04-07T20:56:32.484 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1'] 2024-04-07T20:56:32.484 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:56:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: Upgrade: Target is version 19.0.0-2578-ge5c885fe (unknown) 2024-04-07T20:56:32.759 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1'] 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[24852]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d on smithi187 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: Upgrade: Target is version 19.0.0-2578-ge5c885fe (unknown) 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1'] 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-07T20:56:32.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:32.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-mon[29851]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d on smithi187 2024-04-07T20:56:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-07T20:56:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:32 smithi187 ceph-mon[25907]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d on smithi187 2024-04-07T20:56:33.178 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:32 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:56:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:56:33.946 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:56:33.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:33.946 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:33.946 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:34.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:34.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:34.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:34.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:34.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:34.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:33 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:34.261 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:33 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:34.261 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:34.261 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:33 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:35.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:34 smithi177 ceph-mon[24852]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:35.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:34 smithi177 ceph-mon[29851]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:35.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:34 smithi187 ceph-mon[25907]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:37.258 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:36 smithi177 ceph-mon[29851]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:37.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:36 smithi177 ceph-mon[24852]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:37.281 INFO:teuthology.orchestra.run.smithi177.stdout:true 2024-04-07T20:56:37.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:36 smithi187 ceph-mon[25907]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:agent.smithi177 smithi177 running 14s ago 9m - - 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:agent.smithi187 smithi187 running 4s ago 9m - - 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:alertmanager.a smithi177 *:9093,9094 running (3m) 14s ago 3m 26.1M - ba2b418f427c bdca865a7e70 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:grafana.a smithi187 *:3000 running (3m) 4s ago 3m 54.0M - 8.3.5 dad864ee21e9 7b41716cc5e1 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:iscsi.foo.smithi177.kntdfp smithi177 running (2m) 14s ago 2m 52.5M - 3.5 e1d6a67b021e d2541e320aef 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:mgr.x smithi187 *:8443 running (8m) 4s ago 8m 412M - 17.2.0 e1d6a67b021e a71bc4bae1d4 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:mgr.y smithi177 *:9283 running (10m) 14s ago 10m 465M - 17.2.0 e1d6a67b021e f850f4734a67 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:mon.a smithi177 running (10m) 14s ago 10m 46.4M 2048M 17.2.0 e1d6a67b021e f4208a5ace0f 2024-04-07T20:56:37.860 INFO:teuthology.orchestra.run.smithi177.stdout:mon.b smithi187 running (8m) 4s ago 8m 41.3M 2048M 17.2.0 e1d6a67b021e a091fb21a3a7 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:mon.c smithi177 running (9m) 14s ago 9m 41.5M 2048M 17.2.0 e1d6a67b021e 5c620a76fdbb 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:node-exporter.a smithi177 *:9100 running (4m) 14s ago 4m 16.7M - 1dbe0e931976 001a28e563be 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:node-exporter.b smithi187 *:9100 running (4m) 4s ago 4m 14.9M - 1dbe0e931976 908396a184b0 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.0 smithi177 running (8m) 14s ago 8m 51.0M 4096M 17.2.0 e1d6a67b021e 82e517b977ab 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.1 smithi177 running (7m) 14s ago 7m 51.0M 4096M 17.2.0 e1d6a67b021e 083118d3b511 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.2 smithi177 running (7m) 14s ago 7m 46.8M 4096M 17.2.0 e1d6a67b021e 8bd183701842 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.3 smithi177 running (7m) 14s ago 7m 49.2M 4096M 17.2.0 e1d6a67b021e b782d950f67a 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.4 smithi187 running (6m) 4s ago 6m 50.4M 4096M 17.2.0 e1d6a67b021e c5615f8077c7 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.5 smithi187 running (6m) 4s ago 6m 48.5M 4096M 17.2.0 e1d6a67b021e fca44dee1afa 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.6 smithi187 running (5m) 4s ago 5m 45.8M 4096M 17.2.0 e1d6a67b021e b45cbc92f590 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:osd.7 smithi187 running (5m) 4s ago 5m 49.4M 4096M 17.2.0 e1d6a67b021e bf0628d56383 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:prometheus.a smithi187 *:9095 running (3m) 4s ago 4m 67.7M - 514e6a882f6e ac118aa18f67 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.foo.smithi177.pbcdof smithi177 *:8000 running (3m) 14s ago 3m 68.3M - 17.2.0 e1d6a67b021e 7f8b02c6bd74 2024-04-07T20:56:37.861 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.foo.smithi187.xszzor smithi187 *:8000 running (3m) 4s ago 3m 70.5M - 17.2.0 e1d6a67b021e a43cdb1e8d0b 2024-04-07T20:56:37.862 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.smpl.smithi177.soabjy smithi177 *:80 running (2m) 14s ago 2m 71.8M - 17.2.0 e1d6a67b021e 8fb2c1b4e46f 2024-04-07T20:56:37.862 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.smpl.smithi187.fluayj smithi187 *:80 running (2m) 4s ago 2m 69.6M - 17.2.0 e1d6a67b021e 3e464562784d 2024-04-07T20:56:38.209 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:56:38.209 INFO:teuthology.orchestra.run.smithi177.stdout: "mon": { 2024-04-07T20:56:38.209 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-07T20:56:38.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:38.209 INFO:teuthology.orchestra.run.smithi177.stdout: "mgr": { 2024-04-07T20:56:38.209 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": { 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": {}, 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "rgw": { 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "overall": { 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:56:38.210 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout: "in_progress": true, 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout: "services_complete": [], 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout: "progress": "0/23 daemons upgraded", 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d image on host smithi187" 2024-04-07T20:56:38.505 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:56:38.851 INFO:teuthology.orchestra.run.smithi177.stdout:HEALTH_OK 2024-04-07T20:56:39.173 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:38 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:38] "GET /metrics HTTP/1.1" 200 214928 "" "Prometheus/2.33.4" 2024-04-07T20:56:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-mon[25907]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-mon[25907]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-mon[25907]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-mon[25907]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3925341071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:56:39.459 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/3135934443' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:56:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[29851]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[29851]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[29851]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[29851]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3925341071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:56:39.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/3135934443' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:56:39.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[24852]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[24852]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[24852]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:39.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[24852]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:39.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3925341071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:56:39.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:39 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/3135934443' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:56:39.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:56:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:56:40.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:40 smithi177 ceph-mon[29851]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:40.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:40 smithi177 ceph-mon[24852]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:40.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:40 smithi187 ceph-mon[25907]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:56:41.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:41 smithi177 ceph-mon[24852]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:41.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:41 smithi177 ceph-mon[29851]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:41.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:41 smithi187 ceph-mon[25907]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:43.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:43 smithi177 ceph-mon[24852]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:43.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:43 smithi177 ceph-mon[29851]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:43.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:56:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:56:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:43 smithi187 ceph-mon[25907]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:56:43.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:44.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:44 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:44.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:44.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:44 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:44.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:44 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:44.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:44.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:44 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:44.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:44 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:44.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:44.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:44 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:45.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:45 smithi177 ceph-mon[24852]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:45.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:45 smithi177 ceph-mon[29851]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:45.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:45 smithi187 ceph-mon[25907]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:47.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:47 smithi177 ceph-mon[24852]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:47.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:47 smithi177 ceph-mon[29851]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:47.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:47 smithi187 ceph-mon[25907]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:49.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:49 smithi177 ceph-mon[24852]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:49.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:49 smithi177 ceph-mon[29851]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:49.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:48] "GET /metrics HTTP/1.1" 200 214922 "" "Prometheus/2.33.4" 2024-04-07T20:56:49.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:49 smithi187 ceph-mon[25907]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:49.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:56:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:56:51.449 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:51 smithi177 ceph-mon[24852]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:51.450 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:51 smithi177 ceph-mon[29851]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:51.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:51 smithi187 ceph-mon[25907]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:53.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.187 - - [07/Apr/2024:20:56:53] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:56:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-mon[24852]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:53.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:53.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-mon[29851]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:53.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:53.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:53.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:53 smithi187 ceph-mon[25907]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:53.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:53.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:53 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:56:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:56:53.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:56:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:56:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:56:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:56:54.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:54 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:54.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:54 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:54.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:54 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:56:55.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:55 smithi177 ceph-mon[24852]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:55.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:55 smithi177 ceph-mon[29851]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:55.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:55 smithi187 ceph-mon[25907]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:57.508 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:57 smithi177 ceph-mon[29851]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:57.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:57 smithi177 ceph-mon[24852]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:57.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:57 smithi187 ceph-mon[25907]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:56:59.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:56:59 smithi177 ceph-mon[24852]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:59.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:56:58 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:58] "GET /metrics HTTP/1.1" 200 214930 "" "Prometheus/2.33.4" 2024-04-07T20:56:59.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:56:59 smithi177 ceph-mon[29851]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:59.458 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:56:59 smithi187 ceph-mon[25907]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:56:59.806 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:56:59 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[27323]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:56:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:57:00.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:00 smithi177 ceph-mon[24852]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:00.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:00 smithi177 ceph-mon[29851]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:00.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:00 smithi187 ceph-mon[25907]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:03.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:03 smithi187 ceph-mon[25907]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:03.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:03 smithi177 ceph-mon[29851]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:03.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:03 smithi177 ceph-mon[24852]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:04.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 172.21.15.177 - - [07/Apr/2024:20:57:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-07T20:57:04.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:57:03.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:57:04.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:57:04.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.187 because it doesn't contain any IP SANs" 2024-04-07T20:57:04.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:04 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:04.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:04.309 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:04 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:04.508 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:04 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:04.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:04 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:04 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:04.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:04 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:05.344 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:05 smithi187 ceph-mon[25907]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:05.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:05 smithi177 ceph-mon[24852]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:05.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:05 smithi177 ceph-mon[29851]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: Upgrade: Updating mgr.x 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:57:07.243 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:06 smithi187 ceph-mon[25907]: Deploying daemon mgr.x on smithi187 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: Upgrade: Updating mgr.x 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:57:07.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[24852]: Deploying daemon mgr.x on smithi187 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: Upgrade: Updating mgr.x 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:57:07.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:06 smithi177 ceph-mon[29851]: Deploying daemon mgr.x on smithi187 2024-04-07T20:57:07.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:07 smithi187 systemd[1]: Stopping Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:07.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:07 smithi187 bash[51874]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr.x" found: no such container 2024-04-07T20:57:07.807 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:07 smithi187 podman[51887]: 2024-04-07 20:57:07.681222069 +0000 UTC m=+0.126110090 container died a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, name=centos-stream, distribution-scope=public, ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, release=754, maintainer=Guillaume Abrioux ) 2024-04-07T20:57:08.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 podman[51887]: 2024-04-07 20:57:08.102161932 +0000 UTC m=+0.547049948 container cleanup a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, release=754, distribution-scope=public, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=) 2024-04-07T20:57:08.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 bash[51887]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x 2024-04-07T20:57:08.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 bash[51928]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr.x" found: no such container 2024-04-07T20:57:08.729 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 podman[51902]: 2024-04-07 20:57:08.336884462 +0000 UTC m=+0.647612559 container remove a71bc4bae1d4f33b3e431f678bbb053748925c8ea136d14b8a5d5fd4b4514779 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, architecture=x86_64, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, version=8, com.redhat.component=centos-stream-container) 2024-04-07T20:57:08.729 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-07T20:57:08.730 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.x.service: Failed with result 'exit-code'. 2024-04-07T20:57:08.730 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 systemd[1]: Stopped Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:57:08.730 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.x.service: Consumed 16.714s CPU time. 2024-04-07T20:57:08.730 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 systemd[1]: Starting Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:08.946 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:08 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:08.612Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:09.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 podman[52077]: 2024-04-07 20:57:08.692312288 +0000 UTC m=+0.019580313 image pull fe467d9065a1c6ff757668cf0f97eb2a00e56f51271fffd4b076c4d1580a91ce quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T20:57:09.056 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:08 smithi187 podman[52077]: 2024-04-07 20:57:08.838408889 +0000 UTC m=+0.165676886 container create 9b23674cc983fcfdaa7a929c73685c3514784a04873d0c039410113b3b83705e (image=quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-e5c885f, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-07T20:57:09.174 INFO:teuthology.orchestra.run.smithi177.stdout:true 2024-04-07T20:57:09.234 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:09 smithi177 ceph-mon[29851]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:09.234 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:09 smithi177 ceph-mon[24852]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:09.234 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:57:09] "GET /metrics HTTP/1.1" 200 214930 "" "Prometheus/2.33.4" 2024-04-07T20:57:09.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-mon[25907]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 podman[52077]: 2024-04-07 20:57:09.057584819 +0000 UTC m=+0.384852816 container init 9b23674cc983fcfdaa7a929c73685c3514784a04873d0c039410113b3b83705e (image=quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-e5c885f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 podman[52077]: 2024-04-07 20:57:09.060340564 +0000 UTC m=+0.387608570 container start 9b23674cc983fcfdaa7a929c73685c3514784a04873d0c039410113b3b83705e (image=quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-e5c885f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux ) 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-mgr[52125]: -- 172.21.15.187:0/2973017943 <== mon.2 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556fcc26a5a0 con 0x556fcc23f000 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.118+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.118+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.123+0000 7ff771042200 -1 mgr[py] Class not found in module 'restful' 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.123+0000 7ff771042200 -1 mgr[py] Error loading module 'restful': (22) Invalid argument 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 bash[52077]: 9b23674cc983fcfdaa7a929c73685c3514784a04873d0c039410113b3b83705e 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 systemd[1]: Started Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.141+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.307 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.141+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.146+0000 7ff771042200 -1 mgr[py] Class not found in module 'mds_autoscaler' 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.146+0000 7ff771042200 -1 mgr[py] Error loading module 'mds_autoscaler': (22) Invalid argument 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.166+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.166+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.308 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.173+0000 7ff771042200 -1 mgr[py] Class not found in module 'osd_perf_query' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.173+0000 7ff771042200 -1 mgr[py] Error loading module 'osd_perf_query': (22) Invalid argument 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.191+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.191+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.195+0000 7ff771042200 -1 mgr[py] Class not found in module 'snap_schedule' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.195+0000 7ff771042200 -1 mgr[py] Error loading module 'snap_schedule': (22) Invalid argument 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.211+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.211+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.215+0000 7ff771042200 -1 mgr[py] Class not found in module 'dashboard' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.215+0000 7ff771042200 -1 mgr[py] Error loading module 'dashboard': (22) Invalid argument 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.231+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.231+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.309 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.235+0000 7ff771042200 -1 mgr[py] Class not found in module 'zabbix' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.235+0000 7ff771042200 -1 mgr[py] Error loading module 'zabbix': (22) Invalid argument 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.250+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.250+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.254+0000 7ff771042200 -1 mgr[py] Class not found in module 'balancer' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.254+0000 7ff771042200 -1 mgr[py] Error loading module 'balancer': (22) Invalid argument 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.270+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.270+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.275+0000 7ff771042200 -1 mgr[py] Class not found in module 'influx' 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.275+0000 7ff771042200 -1 mgr[py] Error loading module 'influx': (22) Invalid argument 2024-04-07T20:57:09.310 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.304+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.304+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.308+0000 7ff771042200 -1 mgr[py] Class not found in module 'alerts' 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.308+0000 7ff771042200 -1 mgr[py] Error loading module 'alerts': (22) Invalid argument 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.329+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.329+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.565 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.336+0000 7ff771042200 -1 mgr[py] Class not found in module 'iostat' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.336+0000 7ff771042200 -1 mgr[py] Error loading module 'iostat': (22) Invalid argument 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.351+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.351+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.355+0000 7ff771042200 -1 mgr[py] Class not found in module 'rgw' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.355+0000 7ff771042200 -1 mgr[py] Error loading module 'rgw': (22) Invalid argument 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.370+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.370+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.374+0000 7ff771042200 -1 mgr[py] Class not found in module 'rbd_support' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.374+0000 7ff771042200 -1 mgr[py] Error loading module 'rbd_support': (22) Invalid argument 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.390+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.566 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.390+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.395+0000 7ff771042200 -1 mgr[py] Class not found in module 'progress' 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.395+0000 7ff771042200 -1 mgr[py] Error loading module 'progress': (22) Invalid argument 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.413+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.413+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.567 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.418+0000 7ff771042200 -1 mgr[py] Class not found in module 'pg_autoscaler' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.418+0000 7ff771042200 -1 mgr[py] Error loading module 'pg_autoscaler': (22) Invalid argument 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.439+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.439+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.445+0000 7ff771042200 -1 mgr[py] Class not found in module 'devicehealth' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.445+0000 7ff771042200 -1 mgr[py] Error loading module 'devicehealth': (22) Invalid argument 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.462+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.462+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.466+0000 7ff771042200 -1 mgr[py] Class not found in module 'k8sevents' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.466+0000 7ff771042200 -1 mgr[py] Error loading module 'k8sevents': (22) Invalid argument 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.482+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.482+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.486+0000 7ff771042200 -1 mgr[py] Class not found in module 'rook' 2024-04-07T20:57:09.568 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.486+0000 7ff771042200 -1 mgr[py] Error loading module 'rook': (22) Invalid argument 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.503+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.503+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.507+0000 7ff771042200 -1 mgr[py] Class not found in module 'diskprediction_local' 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.507+0000 7ff771042200 -1 mgr[py] Error loading module 'diskprediction_local': (22) Invalid argument 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.544+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.544+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.549+0000 7ff771042200 -1 mgr[py] Class not found in module 'mirroring' 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.549+0000 7ff771042200 -1 mgr[py] Error loading module 'mirroring': (22) Invalid argument 2024-04-07T20:57:09.569 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.563+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.563+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.567+0000 7ff771042200 -1 mgr[py] Class not found in module 'selftest' 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.567+0000 7ff771042200 -1 mgr[py] Error loading module 'selftest': (22) Invalid argument 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.582+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.582+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.586+0000 7ff771042200 -1 mgr[py] Class not found in module 'telegraf' 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.586+0000 7ff771042200 -1 mgr[py] Error loading module 'telegraf': (22) Invalid argument 2024-04-07T20:57:09.818 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.601+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.601+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.605+0000 7ff771042200 -1 mgr[py] Class not found in module 'test_orchestrator' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.605+0000 7ff771042200 -1 mgr[py] Error loading module 'test_orchestrator': (22) Invalid argument 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.620+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.620+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.624+0000 7ff771042200 -1 mgr[py] Class not found in module 'localpool' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.624+0000 7ff771042200 -1 mgr[py] Error loading module 'localpool': (22) Invalid argument 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.639+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.639+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.643+0000 7ff771042200 -1 mgr[py] Class not found in module 'crash' 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.643+0000 7ff771042200 -1 mgr[py] Error loading module 'crash': (22) Invalid argument 2024-04-07T20:57:09.819 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.658+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.658+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.661+0000 7ff771042200 -1 mgr[py] Class not found in module 'insights' 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.661+0000 7ff771042200 -1 mgr[py] Error loading module 'insights': (22) Invalid argument 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.676+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.676+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.820 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.680+0000 7ff771042200 -1 mgr[py] Class not found in module 'orchestrator' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.680+0000 7ff771042200 -1 mgr[py] Error loading module 'orchestrator': (22) Invalid argument 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.695+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.695+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.699+0000 7ff771042200 -1 mgr[py] Class not found in module 'osd_support' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.699+0000 7ff771042200 -1 mgr[py] Error loading module 'osd_support': (22) Invalid argument 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.715+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.715+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.719+0000 7ff771042200 -1 mgr[py] Class not found in module 'volumes' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.719+0000 7ff771042200 -1 mgr[py] Error loading module 'volumes': (22) Invalid argument 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.773+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.773+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.821 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.777+0000 7ff771042200 -1 mgr[py] Class not found in module 'stats' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.777+0000 7ff771042200 -1 mgr[py] Error loading module 'stats': (22) Invalid argument 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.792+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.792+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.796+0000 7ff771042200 -1 mgr[py] Class not found in module 'cephadm' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.796+0000 7ff771042200 -1 mgr[py] Error loading module 'cephadm': (22) Invalid argument 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.811+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.811+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.816+0000 7ff771042200 -1 mgr[py] Class not found in module 'telemetry' 2024-04-07T20:57:09.822 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.816+0000 7ff771042200 -1 mgr[py] Error loading module 'telemetry': (22) Invalid argument 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:agent.smithi177 smithi177 running 6s ago 10m - - 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:agent.smithi187 smithi187 running 16s ago 9m - - 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:alertmanager.a smithi177 *:9093,9094 running (4m) 6s ago 4m 25.8M - ba2b418f427c bdca865a7e70 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:grafana.a smithi187 *:3000 running (4m) 16s ago 4m 54.0M - 8.3.5 dad864ee21e9 7b41716cc5e1 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:iscsi.foo.smithi177.kntdfp smithi177 running (3m) 6s ago 3m 52.7M - 3.5 e1d6a67b021e d2541e320aef 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:mgr.x smithi187 *:8443,9283 starting - - - - 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:mgr.y smithi177 *:9283 running (11m) 6s ago 11m 466M - 17.2.0 e1d6a67b021e f850f4734a67 2024-04-07T20:57:09.823 INFO:teuthology.orchestra.run.smithi177.stdout:mon.a smithi177 running (11m) 6s ago 11m 47.8M 2048M 17.2.0 e1d6a67b021e f4208a5ace0f 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:mon.b smithi187 running (9m) 16s ago 9m 41.4M 2048M 17.2.0 e1d6a67b021e a091fb21a3a7 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:mon.c smithi177 running (9m) 6s ago 9m 42.9M 2048M 17.2.0 e1d6a67b021e 5c620a76fdbb 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:node-exporter.a smithi177 *:9100 running (5m) 6s ago 5m 16.7M - 1dbe0e931976 001a28e563be 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:node-exporter.b smithi187 *:9100 running (5m) 16s ago 5m 14.9M - 1dbe0e931976 908396a184b0 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.0 smithi177 running (8m) 6s ago 8m 51.4M 4096M 17.2.0 e1d6a67b021e 82e517b977ab 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.1 smithi177 running (8m) 6s ago 8m 51.4M 4096M 17.2.0 e1d6a67b021e 083118d3b511 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.2 smithi177 running (8m) 6s ago 8m 47.0M 4096M 17.2.0 e1d6a67b021e 8bd183701842 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.3 smithi177 running (7m) 6s ago 7m 49.6M 4096M 17.2.0 e1d6a67b021e b782d950f67a 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.4 smithi187 running (7m) 16s ago 7m 50.5M 4096M 17.2.0 e1d6a67b021e c5615f8077c7 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.5 smithi187 running (6m) 16s ago 6m 48.7M 4096M 17.2.0 e1d6a67b021e fca44dee1afa 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.6 smithi187 running (6m) 16s ago 6m 45.9M 4096M 17.2.0 e1d6a67b021e b45cbc92f590 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:osd.7 smithi187 running (6m) 16s ago 6m 49.5M 4096M 17.2.0 e1d6a67b021e bf0628d56383 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:prometheus.a smithi187 *:9095 running (4m) 16s ago 5m 67.7M - 514e6a882f6e ac118aa18f67 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.foo.smithi177.pbcdof smithi177 *:8000 running (3m) 6s ago 3m 68.5M - 17.2.0 e1d6a67b021e 7f8b02c6bd74 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.foo.smithi187.xszzor smithi187 *:8000 running (3m) 16s ago 3m 70.6M - 17.2.0 e1d6a67b021e a43cdb1e8d0b 2024-04-07T20:57:09.824 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.smpl.smithi177.soabjy smithi177 *:80 running (3m) 6s ago 3m 72.0M - 17.2.0 e1d6a67b021e 8fb2c1b4e46f 2024-04-07T20:57:09.825 INFO:teuthology.orchestra.run.smithi177.stdout:rgw.smpl.smithi187.fluayj smithi187 *:80 running (3m) 16s ago 3m 69.7M - 17.2.0 e1d6a67b021e 3e464562784d 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "mon": { 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "mgr": { 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T20:57:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": { 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": {}, 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "rgw": { 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "overall": { 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-07T20:57:10.190 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.832+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.832+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.836+0000 7ff771042200 -1 mgr[py] Class not found in module 'prometheus' 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.836+0000 7ff771042200 -1 mgr[py] Error loading module 'prometheus': (22) Invalid argument 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.852+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.852+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.857+0000 7ff771042200 -1 mgr[py] Class not found in module 'status' 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.857+0000 7ff771042200 -1 mgr[py] Error loading module 'status': (22) Invalid argument 2024-04-07T20:57:10.192 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.872+0000 7ff771042200 -1 mgr[py] Module not found: 'mgr_module' 2024-04-07T20:57:10.193 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.872+0000 7ff771042200 -1 mgr[py] ModuleNotFoundError: No module named 'mgr_module' 2024-04-07T20:57:10.193 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:10.193 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.876+0000 7ff771042200 -1 mgr[py] Class not found in module 'nfs' 2024-04-07T20:57:10.193 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.876+0000 7ff771042200 -1 mgr[py] Error loading module 'nfs': (22) Invalid argument 2024-04-07T20:57:10.193 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:09 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:09.876+0000 7ff771042200 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: restful, mds_autoscaler, osd_perf_query, snap_schedule, dashboard, zabbix, balancer, influx, alerts, iostat, rgw, rbd_support, progress, pg_autoscaler, devicehealth, k8sevents, rook, diskprediction_local, mirroring, selftest, telegraf, test_orchestrator, localpool, crash, insights, orchestrator, osd_support, volumes, stats, cephadm, telemetry, prometheus, status, nfs 2024-04-07T20:57:10.482 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: osdmap e97: 8 total, 8 up, 8 in 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: Standby manager daemon y started 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: Active manager daemon x restarted 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: Activating manager daemon x 2024-04-07T20:57:10.483 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: osdmap e98: 8 total, 8 up, 8 in 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/2104326307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-07T20:57:10.484 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: osdmap e97: 8 total, 8 up, 8 in 2024-04-07T20:57:10.485 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: Standby manager daemon y started 2024-04-07T20:57:10.485 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: Active manager daemon x restarted 2024-04-07T20:57:10.485 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: Activating manager daemon x 2024-04-07T20:57:10.485 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: osdmap e98: 8 total, 8 up, 8 in 2024-04-07T20:57:10.485 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/2104326307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout: "in_progress": true, 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout: "services_complete": [], 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout: "progress": "", 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout: "message": "Currently upgrading mgr daemons" 2024-04-07T20:57:10.500 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-07T20:57:10.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:10.530 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 172.21.15.177:0/359626846' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: osdmap e97: 8 total, 8 up, 8 in 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: Standby manager daemon y started 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: Active manager daemon x restarted 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: Activating manager daemon x 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: osdmap e98: 8 total, 8 up, 8 in 2024-04-07T20:57:10.531 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:10 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/2104326307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T20:57:10.894 INFO:teuthology.orchestra.run.smithi177.stdout:HEALTH_OK 2024-04-07T20:57:11.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setuser ceph since I am not root 2024-04-07T20:57:11.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ignoring --setgroup ceph since I am not root 2024-04-07T20:57:11.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:10 smithi177 ceph-mgr[25089]: -- 172.21.15.177:0/3928763911 <== mon.0 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a396182340 con 0x55a39615b000 2024-04-07T20:57:11.466 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:11.466 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:57:11.466 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: Failing over to other MGR 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:57:11.467 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: mgrmap e21: x(active, starting, since 0.825548s), standbys: y 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: Manager daemon x is now available 2024-04-07T20:57:11.468 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:11 smithi187 ceph-mon[25907]: from='client.? 172.21.15.177:0/1781509213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:11.256+0000 7fc6de2f2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:11.346+0000 7fc6de2f2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: Failing over to other MGR 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:11.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-07T20:57:11.510 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: mgrmap e21: x(active, starting, since 0.825548s), standbys: y 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: Manager daemon x is now available 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[24852]: from='client.? 172.21.15.177:0/1781509213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-07T20:57:11.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: Failing over to other MGR 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T20:57:11.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.14454 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: mgrmap e21: x(active, starting, since 0.825548s), standbys: y 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='mgr.34934 172.21.15.187:0/3484173625' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: Manager daemon x is now available 2024-04-07T20:57:11.513 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-mon[29851]: from='client.? 172.21.15.177:0/1781509213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T20:57:12.005 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:11 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:11.943+0000 7fc6de2f2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-07T20:57:12.258 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:12.006Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:57:12.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.031+0000 7fc6de2f2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-07T20:57:12.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.113+0000 7fc6de2f2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-07T20:57:12.527 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.261+0000 7fc6de2f2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-07T20:57:12.527 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.340+0000 7fc6de2f2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-07T20:57:12.527 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.446+0000 7fc6de2f2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-07T20:57:12.528 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.528+0000 7fc6de2f2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-07T20:57:13.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-mon[29851]: mgrmap e22: x(active, since 1.83453s), standbys: y 2024-04-07T20:57:13.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-mon[29851]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:13.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-mon[24852]: mgrmap e22: x(active, since 1.83453s), standbys: y 2024-04-07T20:57:13.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-mon[24852]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:13.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.690+0000 7fc6de2f2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-07T20:57:13.009 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:12 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:12.774+0000 7fc6de2f2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-07T20:57:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:12 smithi187 ceph-mon[25907]: mgrmap e22: x(active, since 1.83453s), standbys: y 2024-04-07T20:57:13.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:12 smithi187 ceph-mon[25907]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:13.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:13.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-mon[29851]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:13.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-mon[29851]: Health check failed: 16 mgr modules have failed dependencies (MGR_MODULE_DEPENDENCY) 2024-04-07T20:57:13.787 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:13.787 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-mon[24852]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:13.787 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-mon[24852]: Health check failed: 16 mgr modules have failed dependencies (MGR_MODULE_DEPENDENCY) 2024-04-07T20:57:13.787 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:57:13.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:57:13.787 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:57:13.787 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:14.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:13 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:14.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:13 smithi187 ceph-mon[25907]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:14.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:13 smithi187 ceph-mon[25907]: Health check failed: 16 mgr modules have failed dependencies (MGR_MODULE_DEPENDENCY) 2024-04-07T20:57:14.102 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:13.786+0000 7fc6de2f2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-07T20:57:14.102 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:13.914+0000 7fc6de2f2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-07T20:57:14.102 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:14.102+0000 7fc6de2f2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-07T20:57:14.508 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:14.190+0000 7fc6de2f2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-07T20:57:14.509 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:14.324+0000 7fc6de2f2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:57:14.947 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:14.642+0000 7fc6de2f2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-07T20:57:14.947 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:14.863+0000 7fc6de2f2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-07T20:57:15.258 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:14 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:14.948+0000 7fc6de2f2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-07T20:57:15.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:15 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:15.122+0000 7fc6de2f2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-07T20:57:16.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:15 smithi177 ceph-mon[29851]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:16.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:15 smithi177 ceph-mon[24852]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:16.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:15 smithi187 ceph-mon[25907]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:16.355 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:16.061+0000 7fc6de2f2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-07T20:57:16.355 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:16.355+0000 7fc6de2f2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-07T20:57:16.619 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:16.452+0000 7fc6de2f2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-07T20:57:16.619 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:16.604+0000 7fc6de2f2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-07T20:57:16.878 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[24852]: Standby manager daemon y restarted 2024-04-07T20:57:16.878 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[24852]: Standby manager daemon y started 2024-04-07T20:57:16.878 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[24852]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[29851]: Standby manager daemon y restarted 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[29851]: Standby manager daemon y started 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-07T20:57:16.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-mon[29851]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:57:16.880 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:57:16] ENGINE Bus STARTING 2024-04-07T20:57:16.880 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: CherryPy Checker: 2024-04-07T20:57:16.880 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: The Application mounted at '' has an empty config. 2024-04-07T20:57:16.880 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: 2024-04-07T20:57:16.880 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:57:16] ENGINE Serving on http://:::9283 2024-04-07T20:57:16.880 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: [07/Apr/2024:20:57:16] ENGINE Bus STARTED 2024-04-07T20:57:17.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:16 smithi187 ceph-mon[25907]: Standby manager daemon y restarted 2024-04-07T20:57:17.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:16 smithi187 ceph-mon[25907]: Standby manager daemon y started 2024-04-07T20:57:17.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:16 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-07T20:57:17.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:16 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T20:57:17.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:16 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-07T20:57:17.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:16 smithi187 ceph-mon[25907]: from='mgr.? 172.21.15.177:0/1524044865' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T20:57:17.259 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:16 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:16.879Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:57:18.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:17 smithi177 ceph-mon[29851]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:18.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:17 smithi177 ceph-mon[29851]: mgrmap e23: x(active, since 6s), standbys: y 2024-04-07T20:57:18.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:17 smithi177 ceph-mon[24852]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:18.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:17 smithi177 ceph-mon[24852]: mgrmap e23: x(active, since 6s), standbys: y 2024-04-07T20:57:18.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:17 smithi187 ceph-mon[25907]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-07T20:57:18.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:17 smithi187 ceph-mon[25907]: mgrmap e23: x(active, since 6s), standbys: y 2024-04-07T20:57:19.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:57:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:57:20.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:19 smithi177 ceph-mon[24852]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-04-07T20:57:20.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:19 smithi177 ceph-mon[29851]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-04-07T20:57:20.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:19 smithi187 ceph-mon[25907]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-04-07T20:57:21.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:21 smithi187 ceph-mon[25907]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-07T20:57:22.009 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:21 smithi177 ceph-mon[24852]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-07T20:57:22.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:21 smithi177 ceph-mon[29851]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-07T20:57:23.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:23 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:23.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:23 smithi187 ceph-mon[25907]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-04-07T20:57:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:23.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-mon[24852]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-04-07T20:57:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:23.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-mon[29851]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-04-07T20:57:24.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:57:23.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:24.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:23.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:24.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:57:26.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:25 smithi187 ceph-mon[25907]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:26.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:25 smithi177 ceph-mon[24852]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:26.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:25 smithi177 ceph-mon[29851]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:27.962 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:27 smithi177 ceph-mon[29851]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:27.962 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:27 smithi177 ceph-mon[24852]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:28.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:27 smithi187 ceph-mon[25907]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:29.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:28 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:57:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:57:29.987 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:29 smithi177 ceph-mon[24852]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:29.987 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:29 smithi177 ceph-mon[29851]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:30.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:29 smithi187 ceph-mon[25907]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:32.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:31 smithi187 ceph-mon[25907]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:32.258 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:31 smithi177 ceph-mon[24852]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:32.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:31 smithi177 ceph-mon[29851]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:33.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-mon[29851]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:33.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:33.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-mon[24852]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:33.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:33 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:33.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:33 smithi187 ceph-mon[25907]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:57:33.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:33.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:57:36.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:35 smithi187 ceph-mon[25907]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:36.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:35 smithi177 ceph-mon[24852]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:36.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:35 smithi177 ceph-mon[29851]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:38.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:37 smithi187 ceph-mon[25907]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:38.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:37 smithi177 ceph-mon[24852]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:38.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:37 smithi177 ceph-mon[29851]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:39.259 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:38 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:57:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:57:39.914 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:39 smithi187 ceph-mon[25907]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:40.053 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:39 smithi177 ceph-mon[29851]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:40.054 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:39 smithi177 ceph-mon[24852]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg stat 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg getmap 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump [...] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump_json [...] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump_pools_json 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls-by-pool [...] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls-by-primary [] [...] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls-by-osd [] [...] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls [] [...] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump_stuck [...] [] 2024-04-07T20:57:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:Error EINVAL: invalid command 2024-04-07T20:57:41.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch ps' 2024-04-07T20:57:42.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:41 smithi187 ceph-mon[25907]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:42.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:41 smithi177 ceph-mon[29851]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:42.184 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:41 smithi177 ceph-mon[24852]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-07T20:57:43.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-mon[29851]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:43.509 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-mon[29851]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-mon[24852]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:43.509 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-mon[24852]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:43.556 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:43 smithi187 ceph-mon[25907]: from='client.14961 -' entity='client.iscsi.foo.smithi177.kntdfp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-07T20:57:43.557 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:43 smithi187 ceph-mon[25907]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-07T20:57:43.785 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:57:43.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:43.786 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:43.786 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg stat 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg getmap 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump [...] 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump_json [...] 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump_pools_json 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls-by-pool [...] 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls-by-primary [] [...] 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls-by-osd [] [...] 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg ls [] [...] 2024-04-07T20:57:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:pg dump_stuck [...] [] 2024-04-07T20:57:43.939 INFO:teuthology.orchestra.run.smithi177.stderr:Error EINVAL: invalid command 2024-04-07T20:57:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-07T20:57:44.707 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 1459, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch ps'" 2024-04-07T20:57:44.940 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a70562b49cb74f9ba9294ced52b5216b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 1459, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch ps'" 2024-04-07T20:57:44.942 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-07T20:57:44.951 INFO:tasks.cephadm:Teardown begin 2024-04-07T20:57:44.952 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T20:57:44.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T20:57:45.025 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-07T20:57:45.026 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-07T20:57:45.043 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-07T20:57:45.084 INFO:tasks.cephadm:Stopping all daemons... 2024-04-07T20:57:45.084 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-07T20:57:45.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a 2024-04-07T20:57:45.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 systemd[1]: Stopping Ceph mon.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:45.259 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 bash[62349]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.a" found: no such container 2024-04-07T20:57:45.595 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a[24848]: 2024-04-07T20:57:45.277+0000 7f6eaf839700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:57:45.595 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a[24848]: 2024-04-07T20:57:45.277+0000 7f6eaf839700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-07T20:57:45.595 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 podman[62362]: 2024-04-07 20:57:45.318867911 +0000 UTC m=+0.159224321 container died f4208a5ace0f8ced6ed50662f402915d4d4ed2911ecb90c7f23e5a927af41a91 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, distribution-scope=public, io.openshift.expose-services=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, version=8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-07T20:57:45.888 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 podman[62362]: 2024-04-07 20:57:45.786897434 +0000 UTC m=+0.627253840 container cleanup f4208a5ace0f8ced6ed50662f402915d4d4ed2911ecb90c7f23e5a927af41a91 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a, release=754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, version=8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux ) 2024-04-07T20:57:45.888 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 bash[62362]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-a 2024-04-07T20:57:45.888 INFO:journalctl@ceph.mon.a.smithi177.stdout:Apr 07 20:57:45 smithi177 bash[62391]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.a" found: no such container 2024-04-07T20:57:46.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.a.service' 2024-04-07T20:57:46.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:57:46.131 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-07T20:57:46.131 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-07T20:57:46.132 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.c 2024-04-07T20:57:46.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 systemd[1]: Stopping Ceph mon.c for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:46.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 bash[62446]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.c" found: no such container 2024-04-07T20:57:46.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-c[29825]: 2024-04-07T20:57:46.316+0000 7fc2ca8a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:57:46.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-c[29825]: 2024-04-07T20:57:46.316+0000 7fc2ca8a6700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-07T20:57:46.512 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 podman[62457]: 2024-04-07 20:57:46.367615431 +0000 UTC m=+0.157882100 container died 5c620a76fdbb30d25bcecd0c71399bbd0a0a066f00940a88c103d455e60745c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-c, release=754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, distribution-scope=public, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-04-07T20:57:47.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 podman[62457]: 2024-04-07 20:57:46.834019192 +0000 UTC m=+0.624285857 container cleanup 5c620a76fdbb30d25bcecd0c71399bbd0a0a066f00940a88c103d455e60745c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-c, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-07T20:57:47.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 bash[62457]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-c 2024-04-07T20:57:47.009 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 07 20:57:46 smithi177 bash[62483]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.c" found: no such container 2024-04-07T20:57:47.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.c.service' 2024-04-07T20:57:47.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:57:47.157 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-07T20:57:47.158 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-07T20:57:47.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.b 2024-04-07T20:57:47.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 systemd[1]: Stopping Ceph mon.b for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:47.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 bash[53092]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.b" found: no such container 2024-04-07T20:57:47.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-b[25903]: 2024-04-07T20:57:47.370+0000 7f66f9aa2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:57:47.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-b[25903]: 2024-04-07T20:57:47.370+0000 7f66f9aa2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-07T20:57:47.796 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 podman[53105]: 2024-04-07 20:57:47.503756459 +0000 UTC m=+0.255946516 container died a091fb21a3a77c6c0df2aabe14dd8cce9b6a3b1d6dd613ebec5a2c5d2bbc64c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, release=754, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git) 2024-04-07T20:57:48.056 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 podman[53105]: 2024-04-07 20:57:47.879225734 +0000 UTC m=+0.631415779 container cleanup a091fb21a3a77c6c0df2aabe14dd8cce9b6a3b1d6dd613ebec5a2c5d2bbc64c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-b, io.buildah.version=1.19.8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T20:57:48.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 bash[53105]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon-b 2024-04-07T20:57:48.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Apr 07 20:57:47 smithi187 bash[53130]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mon.b" found: no such container 2024-04-07T20:57:48.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mon.b.service' 2024-04-07T20:57:48.240 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:57:48.240 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-07T20:57:48.241 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-07T20:57:48.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y 2024-04-07T20:57:48.463 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:48 smithi177 systemd[1]: Stopping Ceph mgr.y for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:49.175 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:48 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y[25085]: ::ffff:172.21.15.187 - - [07/Apr/2024:20:57:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-07T20:57:49.478 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:49 smithi177 bash[62557]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr.y" found: no such container 2024-04-07T20:57:50.239 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:49 smithi177 podman[62601]: 2024-04-07 20:57:49.813695065 +0000 UTC m=+0.562547950 container died f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y, release=754, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_BRANCH=HEAD) 2024-04-07T20:57:50.489 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:50 smithi177 podman[62601]: 2024-04-07 20:57:50.292131868 +0000 UTC m=+1.040984756 container cleanup f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, version=8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T20:57:50.490 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:50 smithi177 bash[62601]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y 2024-04-07T20:57:50.490 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:50 smithi177 podman[62627]: 2024-04-07 20:57:50.468645811 +0000 UTC m=+0.654926977 container remove f850f4734a6731bc01c1d05ae3bf602eb9eaf645dc5be611d370cdc38c4b9eb3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-y, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, version=8, GIT_CLEAN=True, ceph=True, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, release=754) 2024-04-07T20:57:50.490 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 07 20:57:50 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-07T20:57:51.137 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:50 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:50.785Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:57:51.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.y.service' 2024-04-07T20:57:51.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:57:51.204 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-07T20:57:51.204 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-07T20:57:51.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.x 2024-04-07T20:57:51.502 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:51 smithi187 systemd[1]: Stopping Ceph mgr.x for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:51.503 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:51 smithi187 bash[53388]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr.x" found: no such container 2024-04-07T20:57:52.136 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:51 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:51.852+0000 7ff765630700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:57:52.137 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:51 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x[52121]: 2024-04-07T20:57:51.852+0000 7ff765630700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-07T20:57:52.137 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:51 smithi187 podman[53406]: 2024-04-07 20:57:51.864224613 +0000 UTC m=+0.451299489 container died 9b23674cc983fcfdaa7a929c73685c3514784a04873d0c039410113b3b83705e (image=quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-e5c885f, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-07T20:57:52.529 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:52 smithi187 podman[53406]: 2024-04-07 20:57:52.295667931 +0000 UTC m=+0.882742837 container cleanup 9b23674cc983fcfdaa7a929c73685c3514784a04873d0c039410113b3b83705e (image=quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-e5c885f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-07T20:57:52.529 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:52 smithi187 bash[53406]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr-x 2024-04-07T20:57:52.530 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Apr 07 20:57:52 smithi187 bash[53464]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-mgr.x" found: no such container 2024-04-07T20:57:52.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@mgr.x.service' 2024-04-07T20:57:52.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:57:52.577 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-07T20:57:52.577 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-07T20:57:52.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.0 2024-04-07T20:57:53.009 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:52 smithi177 systemd[1]: Stopping Ceph osd.0 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:57:53.009 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:52 smithi177 bash[62738]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.0" found: no such container 2024-04-07T20:57:53.009 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0[33080]: 2024-04-07T20:57:52.775+0000 7f4b88609700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:57:53.010 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0[33080]: 2024-04-07T20:57:52.775+0000 7f4b88609700 -1 osd.0 98 *** Got signal Terminated *** 2024-04-07T20:57:53.010 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:52 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0[33080]: 2024-04-07T20:57:52.775+0000 7f4b88609700 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:57:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:57:53.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:57:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:57:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:57:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:57:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:57:58.476 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:58 smithi177 podman[62751]: 2024-04-07 20:57:58.215431403 +0000 UTC m=+5.549642133 container died 82e517b977ab56951d40e8c261feb81b756260c22b645f5d5858720ef078960b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, distribution-scope=public, io.openshift.tags=base centos centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, ceph=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T20:57:58.759 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:58 smithi177 podman[62751]: 2024-04-07 20:57:58.68401244 +0000 UTC m=+6.018223171 container cleanup 82e517b977ab56951d40e8c261feb81b756260c22b645f5d5858720ef078960b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=centos-stream-container, release=754, ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, version=8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T20:57:58.759 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:58 smithi177 bash[62751]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0 2024-04-07T20:57:58.759 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:58 smithi177 bash[62781]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.0" found: no such container 2024-04-07T20:57:59.233 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:58 smithi177 podman[62769]: 2024-04-07 20:57:58.935627556 +0000 UTC m=+0.713008982 container remove 82e517b977ab56951d40e8c261feb81b756260c22b645f5d5858720ef078960b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, distribution-scope=public, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream) 2024-04-07T20:57:59.233 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:59 smithi177 podman[62848]: 2024-04-07 20:57:59.021533712 +0000 UTC m=+0.019414457 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:57:59.233 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:59 smithi177 podman[62848]: 2024-04-07 20:57:59.128231973 +0000 UTC m=+0.126112733 container create a4029ad3bc36e0c192c2296b76d649a09c55277b7e837123edb7c18303298ee4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, release=754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8) 2024-04-07T20:57:59.509 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:59 smithi177 podman[62848]: 2024-04-07 20:57:59.33913071 +0000 UTC m=+0.337011434 container init a4029ad3bc36e0c192c2296b76d649a09c55277b7e837123edb7c18303298ee4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0-deactivate, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8, name=centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, ceph=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8) 2024-04-07T20:57:59.509 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:59 smithi177 podman[62848]: 2024-04-07 20:57:59.342484108 +0000 UTC m=+0.340364832 container start a4029ad3bc36e0c192c2296b76d649a09c55277b7e837123edb7c18303298ee4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0-deactivate, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_CLEAN=True, io.openshift.expose-services=, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-04-07T20:57:59.510 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:59 smithi177 podman[62848]: 2024-04-07 20:57:59.422229777 +0000 UTC m=+0.420110505 container attach a4029ad3bc36e0c192c2296b76d649a09c55277b7e837123edb7c18303298ee4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0-deactivate, io.openshift.tags=base centos centos-stream, name=centos-stream, RELEASE=HEAD, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=754, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc.) 2024-04-07T20:57:59.899 INFO:journalctl@ceph.osd.0.smithi177.stdout:Apr 07 20:57:59 smithi177 podman[62848]: 2024-04-07 20:57:59.620385784 +0000 UTC m=+0.618266513 container died a4029ad3bc36e0c192c2296b76d649a09c55277b7e837123edb7c18303298ee4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-0-deactivate, name=centos-stream, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vendor=Red Hat, Inc., release=754, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0) 2024-04-07T20:58:00.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-07T20:58:00.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:00.256 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-07T20:58:00.256 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-07T20:58:00.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.1 2024-04-07T20:58:00.465 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:00 smithi177 systemd[1]: Stopping Ceph osd.1 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:00.465 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:00 smithi177 bash[62954]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.1" found: no such container 2024-04-07T20:58:00.758 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1[35976]: 2024-04-07T20:58:00.470+0000 7f95ae97b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:00.759 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1[35976]: 2024-04-07T20:58:00.470+0000 7f95ae97b700 -1 osd.1 98 *** Got signal Terminated *** 2024-04-07T20:58:00.759 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:00 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1[35976]: 2024-04-07T20:58:00.470+0000 7f95ae97b700 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:03.806 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:03.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:03.806 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:03.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:03.806 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:03 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:03.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:06.219 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:05 smithi177 podman[62971]: 2024-04-07 20:58:05.917526962 +0000 UTC m=+5.544339457 container died 083118d3b51162fa073207bb2fbb9ee3b0a0ae3a4568d803cdaab4a23728614f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T20:58:06.509 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:06 smithi177 podman[62971]: 2024-04-07 20:58:06.393653174 +0000 UTC m=+6.020465707 container cleanup 083118d3b51162fa073207bb2fbb9ee3b0a0ae3a4568d803cdaab4a23728614f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, release=754, name=centos-stream, version=8) 2024-04-07T20:58:06.509 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:06 smithi177 bash[62971]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1 2024-04-07T20:58:06.509 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:06 smithi177 bash[63622]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.1" found: no such container 2024-04-07T20:58:07.000 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:06 smithi177 podman[63607]: 2024-04-07 20:58:06.670066179 +0000 UTC m=+0.748862578 container remove 083118d3b51162fa073207bb2fbb9ee3b0a0ae3a4568d803cdaab4a23728614f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.expose-services=, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , release=754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T20:58:07.000 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:06 smithi177 podman[63687]: 2024-04-07 20:58:06.792430933 +0000 UTC m=+0.019079812 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:07.000 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:06 smithi177 podman[63687]: 2024-04-07 20:58:06.896146247 +0000 UTC m=+0.122795163 container create b99294c44147fce07604cbf9f5499fdab12a449e547db94e552872c02ab8392b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1-deactivate, vcs-type=git, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T20:58:07.259 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:07 smithi177 podman[63687]: 2024-04-07 20:58:07.106887451 +0000 UTC m=+0.333536322 container init b99294c44147fce07604cbf9f5499fdab12a449e547db94e552872c02ab8392b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vendor=Red Hat, Inc., name=centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, RELEASE=HEAD) 2024-04-07T20:58:07.259 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:07 smithi177 podman[63687]: 2024-04-07 20:58:07.110145168 +0000 UTC m=+0.336794039 container start b99294c44147fce07604cbf9f5499fdab12a449e547db94e552872c02ab8392b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1-deactivate, maintainer=Guillaume Abrioux , release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, name=centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-04-07T20:58:07.259 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:07 smithi177 podman[63687]: 2024-04-07 20:58:07.181762645 +0000 UTC m=+0.408411531 container attach b99294c44147fce07604cbf9f5499fdab12a449e547db94e552872c02ab8392b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1-deactivate, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, RELEASE=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.expose-services=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-04-07T20:58:07.700 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 07 20:58:07 smithi177 podman[63687]: 2024-04-07 20:58:07.387887045 +0000 UTC m=+0.614535917 container died b99294c44147fce07604cbf9f5499fdab12a449e547db94e552872c02ab8392b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-1-deactivate, io.openshift.expose-services=, vcs-type=git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, RELEASE=HEAD, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-07T20:58:08.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-07T20:58:08.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:08.155 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-07T20:58:08.156 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-07T20:58:08.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.2 2024-04-07T20:58:08.392 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:08 smithi177 systemd[1]: Stopping Ceph osd.2 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:08.758 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:08 smithi177 bash[63780]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.2" found: no such container 2024-04-07T20:58:09.637 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2[38909]: 2024-04-07T20:58:09.481+0000 7f0c62336700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:09.637 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2[38909]: 2024-04-07T20:58:09.481+0000 7f0c62336700 -1 osd.2 98 *** Got signal Terminated *** 2024-04-07T20:58:09.637 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:09 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2[38909]: 2024-04-07T20:58:09.481+0000 7f0c62336700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:13.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:14.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:13 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:15.211 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:14 smithi177 podman[63797]: 2024-04-07 20:58:14.953170578 +0000 UTC m=+6.549518740 container died 8bd18370184283e2a3b69fc4a72c86b0e6b2947ab97ce29340dfb3951b534b11 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-04-07T20:58:15.509 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:15 smithi177 podman[63797]: 2024-04-07 20:58:15.386051181 +0000 UTC m=+6.982399343 container cleanup 8bd18370184283e2a3b69fc4a72c86b0e6b2947ab97ce29340dfb3951b534b11 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., release=754, distribution-scope=public, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T20:58:15.509 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:15 smithi177 bash[63797]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2 2024-04-07T20:58:15.924 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:15 smithi177 podman[63906]: 2024-04-07 20:58:15.579075597 +0000 UTC m=+0.624185807 container remove 8bd18370184283e2a3b69fc4a72c86b0e6b2947ab97ce29340dfb3951b534b11 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.19.8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, name=centos-stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8) 2024-04-07T20:58:15.924 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:15 smithi177 bash[63918]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.2" found: no such container 2024-04-07T20:58:15.924 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:15 smithi177 podman[63984]: 2024-04-07 20:58:15.82160313 +0000 UTC m=+0.114892189 container create b506d4e3b996835899db79fde25ad3c131ca66a5e3ad91fc866bd11c3204e3b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T20:58:15.925 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:15 smithi177 podman[63984]: 2024-04-07 20:58:15.725099122 +0000 UTC m=+0.018388182 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:16.259 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:16 smithi177 podman[63984]: 2024-04-07 20:58:16.049069 +0000 UTC m=+0.342358059 container init b506d4e3b996835899db79fde25ad3c131ca66a5e3ad91fc866bd11c3204e3b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, release=754, com.redhat.component=centos-stream-container, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, RELEASE=HEAD, distribution-scope=public) 2024-04-07T20:58:16.259 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:16 smithi177 podman[63984]: 2024-04-07 20:58:16.0520713 +0000 UTC m=+0.345360363 container start b506d4e3b996835899db79fde25ad3c131ca66a5e3ad91fc866bd11c3204e3b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2-deactivate, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0) 2024-04-07T20:58:16.259 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:16 smithi177 podman[63984]: 2024-04-07 20:58:16.132392305 +0000 UTC m=+0.425681375 container attach b506d4e3b996835899db79fde25ad3c131ca66a5e3ad91fc866bd11c3204e3b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True) 2024-04-07T20:58:16.637 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 07 20:58:16 smithi177 podman[63984]: 2024-04-07 20:58:16.334562563 +0000 UTC m=+0.627851637 container died b506d4e3b996835899db79fde25ad3c131ca66a5e3ad91fc866bd11c3204e3b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-2-deactivate, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, release=754, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T20:58:17.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-07T20:58:17.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:17.051 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-07T20:58:17.051 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-07T20:58:17.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.3 2024-04-07T20:58:17.245 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:17 smithi177 systemd[1]: Stopping Ceph osd.3 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:17.245 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:17 smithi177 bash[64063]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.3" found: no such container 2024-04-07T20:58:17.509 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:17 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:17.275+0000 7fa8712bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:17.509 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:17 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:17.275+0000 7fa8712bb700 -1 osd.3 98 *** Got signal Terminated *** 2024-04-07T20:58:17.509 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:17 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:17.275+0000 7fa8712bb700 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:19.259 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:18 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:18.817+0000 7fa86b451700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:54.844015+0000 front 2024-04-07T20:57:54.843787+0000 (oldest deadline 2024-04-07T20:58:18.343958+0000) 2024-04-07T20:58:20.259 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:19 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:19.786+0000 7fa86b451700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:54.844015+0000 front 2024-04-07T20:57:54.843787+0000 (oldest deadline 2024-04-07T20:58:18.343958+0000) 2024-04-07T20:58:20.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:19 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:19.961+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:20.981 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:20 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:20.702+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:21.259 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:20 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:20.815+0000 7fa86b451700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:54.844015+0000 front 2024-04-07T20:57:54.843787+0000 (oldest deadline 2024-04-07T20:58:18.343958+0000) 2024-04-07T20:58:21.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:20 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:20.981+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:21.979 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:21 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:21.724+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:22.009 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:21 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3[41898]: 2024-04-07T20:58:21.807+0000 7fa86b451700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:54.844015+0000 front 2024-04-07T20:57:54.843787+0000 (oldest deadline 2024-04-07T20:58:18.343958+0000) 2024-04-07T20:58:22.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:21 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:21.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:22.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:21 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:21.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:22.306 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:22.082+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:22.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:21 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:21.979+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:22.806 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:22.693+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:22.807 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:22 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:22.506+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:23.042 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:22 smithi177 podman[64076]: 2024-04-07 20:58:22.785382775 +0000 UTC m=+5.611561527 container died b782d950f67a679f61dea36a8c7709f784a82b6167b28c587cfc73df4a192b99 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vendor=Red Hat, Inc.) 2024-04-07T20:58:23.306 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:23.110+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:23.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:23.004+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:23.509 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64076]: 2024-04-07 20:58:23.242071959 +0000 UTC m=+6.068250710 container cleanup b782d950f67a679f61dea36a8c7709f784a82b6167b28c587cfc73df4a192b99 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, architecture=x86_64, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=) 2024-04-07T20:58:23.509 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 bash[64076]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3 2024-04-07T20:58:23.509 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 bash[64118]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.3" found: no such container 2024-04-07T20:58:23.800 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:23.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:23.800 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:23.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:23.800 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:23 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:23.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:23.800 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64104]: 2024-04-07 20:58:23.526680751 +0000 UTC m=+0.736042510 container remove b782d950f67a679f61dea36a8c7709f784a82b6167b28c587cfc73df4a192b99 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T20:58:23.800 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64182]: 2024-04-07 20:58:23.584554177 +0000 UTC m=+0.018579423 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:23.801 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64182]: 2024-04-07 20:58:23.711155444 +0000 UTC m=+0.145180691 container create 8c35886a31371b51b7b0807cbb9c5ac22b483909d17f5cbe7428611165f4a80f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vendor=Red Hat, Inc., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.openshift.expose-services=) 2024-04-07T20:58:23.806 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:23.501+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:23.807 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:23 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:23.669+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:24.065 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:24 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:24.038+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:24.190 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64182]: 2024-04-07 20:58:23.897247393 +0000 UTC m=+0.331272640 container init 8c35886a31371b51b7b0807cbb9c5ac22b483909d17f5cbe7428611165f4a80f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3-deactivate, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, version=8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., ceph=True, release=754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-04-07T20:58:24.191 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64182]: 2024-04-07 20:58:23.900246465 +0000 UTC m=+0.334271712 container start 8c35886a31371b51b7b0807cbb9c5ac22b483909d17f5cbe7428611165f4a80f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, name=centos-stream, release=754, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux ) 2024-04-07T20:58:24.191 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:23 smithi177 podman[64182]: 2024-04-07 20:58:23.980264432 +0000 UTC m=+0.414289695 container attach 8c35886a31371b51b7b0807cbb9c5ac22b483909d17f5cbe7428611165f4a80f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3-deactivate, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, ceph=True, version=8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, RELEASE=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True) 2024-04-07T20:58:24.387 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:24 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:24.065+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:24.507 INFO:journalctl@ceph.osd.3.smithi177.stdout:Apr 07 20:58:24 smithi177 podman[64182]: 2024-04-07 20:58:24.198378158 +0000 UTC m=+0.632403409 container died 8c35886a31371b51b7b0807cbb9c5ac22b483909d17f5cbe7428611165f4a80f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, RELEASE=HEAD, GIT_CLEAN=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T20:58:24.660 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:24 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:24.469+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:24.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-07T20:58:24.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:24.848 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-07T20:58:24.848 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-07T20:58:24.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.4 2024-04-07T20:58:24.919 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:24 smithi187 systemd[1]: Stopping Ceph osd.4 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:24.920 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:24 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:24.661+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:25.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:25.066+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:25.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:25.089+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:25.307 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:24 smithi187 bash[54627]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.4" found: no such container 2024-04-07T20:58:25.307 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:25.028+0000 7fe23eced700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:25.307 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:25.028+0000 7fe23eced700 -1 osd.4 98 *** Got signal Terminated *** 2024-04-07T20:58:25.308 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:25.028+0000 7fe23eced700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:25.695 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:25.438+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:26.056 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:25 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:25.696+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:26.449 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:26.114+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:26.449 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:26.082+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:26.449 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:26.082+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:26.741 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:26.449+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:26.741 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:26.449+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:01.576832+0000 front 2024-04-07T20:58:01.576736+0000 (oldest deadline 2024-04-07T20:58:26.177142+0000) 2024-04-07T20:58:27.056 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:26 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:26.741+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:27.416 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:27.131+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:27.416 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:27.117+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:27.417 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:27.117+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:27.754 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:27.416+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:27.754 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:27.416+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:01.576832+0000 front 2024-04-07T20:58:01.576736+0000 (oldest deadline 2024-04-07T20:58:26.177142+0000) 2024-04-07T20:58:28.056 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:27 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:27.754+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:28.421 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:28.081+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:28.421 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:28.081+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:28.422 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:28.149+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:28.422 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:28.149+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:28.716 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:28.421+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:28.716 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:28.421+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:01.576832+0000 front 2024-04-07T20:58:01.576736+0000 (oldest deadline 2024-04-07T20:58:26.177142+0000) 2024-04-07T20:58:29.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:28.716+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:29.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:28 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:28.716+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:29.306 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:29.183+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:29.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:29.183+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:29.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:29.054+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:29.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:29.054+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:29.718 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:29.426+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.275962+0000 front 2024-04-07T20:57:56.275989+0000 (oldest deadline 2024-04-07T20:58:21.576186+0000) 2024-04-07T20:58:29.719 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4[30418]: 2024-04-07T20:58:29.426+0000 7fe23be89700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:01.576832+0000 front 2024-04-07T20:58:01.576736+0000 (oldest deadline 2024-04-07T20:58:26.177142+0000) 2024-04-07T20:58:30.014 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:29.719+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:30.014 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:29 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:29.719+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:30.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:30 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:30.171+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:30.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:30 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:30.171+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:30.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:30 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:30.014+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:30.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:30 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:30.014+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:30.807 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:30 smithi187 podman[54640]: 2024-04-07 20:58:30.490553256 +0000 UTC m=+5.560604287 container died c5615f8077c76b8b0f1adde7512940b17fc55d17e27bd514d25a8f2445e32a10 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, maintainer=Guillaume Abrioux , release=754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=) 2024-04-07T20:58:30.807 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:30 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:30.692+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:30.807 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:30 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:30.692+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:31.163 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:31 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:31.056+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:31.163 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:31 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:31.056+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:31.163 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:30 smithi187 podman[54640]: 2024-04-07 20:58:30.904157718 +0000 UTC m=+5.974208761 container cleanup c5615f8077c76b8b0f1adde7512940b17fc55d17e27bd514d25a8f2445e32a10 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, version=8, name=centos-stream, GIT_CLEAN=True, architecture=x86_64, distribution-scope=public, build-date=2022-05-03T08:36:31.336870) 2024-04-07T20:58:31.163 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:30 smithi187 bash[54640]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4 2024-04-07T20:58:31.164 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:30 smithi187 bash[54670]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.4" found: no such container 2024-04-07T20:58:31.164 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54657]: 2024-04-07 20:58:31.12193304 +0000 UTC m=+0.624423068 container remove c5615f8077c76b8b0f1adde7512940b17fc55d17e27bd514d25a8f2445e32a10 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, release=754, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream) 2024-04-07T20:58:31.463 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:31 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:31.160+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:31.463 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:31 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:31.160+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:31.463 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54734]: 2024-04-07 20:58:31.252126996 +0000 UTC m=+0.019039289 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:31.463 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54734]: 2024-04-07 20:58:31.356417669 +0000 UTC m=+0.123329987 container create a5dcaa43811d6fe68886f67ffde505d2d3cacae7c105b1c96258e8ab8ddaf13e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-type=git, release=754, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-07T20:58:31.725 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54734]: 2024-04-07 20:58:31.566883109 +0000 UTC m=+0.333795381 container init a5dcaa43811d6fe68886f67ffde505d2d3cacae7c105b1c96258e8ab8ddaf13e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.buildah.version=1.19.8, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-type=git, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD) 2024-04-07T20:58:31.726 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54734]: 2024-04-07 20:58:31.570154591 +0000 UTC m=+0.337066860 container start a5dcaa43811d6fe68886f67ffde505d2d3cacae7c105b1c96258e8ab8ddaf13e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4-deactivate, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, architecture=x86_64, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.component=centos-stream-container, version=8) 2024-04-07T20:58:31.726 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54734]: 2024-04-07 20:58:31.65029577 +0000 UTC m=+0.417208052 container attach a5dcaa43811d6fe68886f67ffde505d2d3cacae7c105b1c96258e8ab8ddaf13e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-07T20:58:31.910 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:31 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:31.626Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:31.910 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:31 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:31.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:31.910 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:31 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:31.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:32.048 INFO:journalctl@ceph.osd.4.smithi187.stdout:Apr 07 20:58:31 smithi187 podman[54734]: 2024-04-07 20:58:31.851326527 +0000 UTC m=+0.618238801 container died a5dcaa43811d6fe68886f67ffde505d2d3cacae7c105b1c96258e8ab8ddaf13e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-4-deactivate, build-date=2022-05-03T08:36:31.336870, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, version=8, name=centos-stream, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T20:58:32.049 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:31 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:31.725+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:32.049 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:31 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:31.725+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:32.306 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:32.171+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:32.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:32.171+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:32.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:32.049+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:32.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:32.049+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:32.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-07T20:58:32.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:32.527 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-07T20:58:32.527 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-07T20:58:32.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.5 2024-04-07T20:58:32.721 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 systemd[1]: Stopping Ceph osd.5 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:32.721 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 bash[54814]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.5" found: no such container 2024-04-07T20:58:32.721 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:32.692+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:32.721 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:32.692+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:33.051 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:32.782+0000 7f91a40a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:33.051 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:32.782+0000 7f91a40a8700 -1 osd.5 98 *** Got signal Terminated *** 2024-04-07T20:58:33.052 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:32 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:32.782+0000 7f91a40a8700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:33.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:33 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:33.142+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:33.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:33 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:33.142+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:33.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:33 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:33.051+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:33.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:33 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:33.051+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:33.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:33.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:34.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:33 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:33.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:34.048 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:33 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:33.726+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:34.049 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:33 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:33.726+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:34.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:34 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:34.049+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:34.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:34 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:34.049+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:34.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:34 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:34.123+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:34.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:34 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:34.123+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:35.003 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:34 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:34.678+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:35.004 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:34 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:34.678+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:35.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:35.004+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:35.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:35.004+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:35.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:35.077+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:35.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:35.077+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:35.963 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:35.685+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:35.963 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:35.685+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:36.290 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:36.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:36.290 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:36.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:36.290 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:36.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:36.291 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:35.963+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:36.291 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:35 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:35.963+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:37.056 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:36.984+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:37.057 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:36.984+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:37.057 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:36.734+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:37.057 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:36 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:36.734+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:37.519 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:37 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:37.119+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:37.519 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:37 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:37.119+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:37.520 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:37 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:37.119+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:37.806 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:37 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:37.739+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.373732+0000 front 2024-04-07T20:57:56.373716+0000 (oldest deadline 2024-04-07T20:58:20.473661+0000) 2024-04-07T20:58:37.807 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:37 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5[33313]: 2024-04-07T20:58:37.739+0000 7f919ea3f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:05.075087+0000 front 2024-04-07T20:58:05.075163+0000 (oldest deadline 2024-04-07T20:58:27.975241+0000) 2024-04-07T20:58:38.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:38 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:38.100+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:38.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:38 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:38.100+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:38.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:38 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:38.100+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:38.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:38 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:38.006+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:38.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:38 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:38.006+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:38.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:38 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:38.006+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:38.652 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:38 smithi187 podman[54827]: 2024-04-07 20:58:38.362677823 +0000 UTC m=+5.714760051 container died fca44dee1afa5741011f38d87f3478266ddfeab0eef1080c95c78611fc6c6b89 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, version=8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T20:58:39.003 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:38 smithi187 podman[54827]: 2024-04-07 20:58:38.827648244 +0000 UTC m=+6.179730464 container cleanup fca44dee1afa5741011f38d87f3478266ddfeab0eef1080c95c78611fc6c6b89 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=) 2024-04-07T20:58:39.003 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:38 smithi187 bash[54827]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5 2024-04-07T20:58:39.306 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:39 smithi187 podman[55308]: 2024-04-07 20:58:39.003853933 +0000 UTC m=+0.742970191 container remove fca44dee1afa5741011f38d87f3478266ddfeab0eef1080c95c78611fc6c6b89 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., ceph=True, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T20:58:39.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:39.140+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:39.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:39.140+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:39.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:39.140+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:39.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:39.029+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:39.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:39.029+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:39.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:39.029+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:39.723 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:39 smithi187 bash[55344]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.5" found: no such container 2024-04-07T20:58:40.258 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:39.996+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:40.259 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:39.996+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:40.259 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:39 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:39.996+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:40.259 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:40.114+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:40.259 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:40.114+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:40.260 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:40.114+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:41.148 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:40.964+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:41.148 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:40.964+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:41.148 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:40.964+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:41.149 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:41 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:41.094+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:41.149 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:41 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:41.094+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:41.149 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:41 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:41.094+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:41.149 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:40 smithi187 podman[55570]: 2024-04-07 20:58:40.782005525 +0000 UTC m=+0.020022029 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:41.149 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:40 smithi187 podman[55570]: 2024-04-07 20:58:40.896629595 +0000 UTC m=+0.134646134 container create 821ef5ab9ce7386a3ede657222fd53e59e006c5ec22172b737a9053c33f0035b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5-deactivate, vcs-type=git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , release=754, name=centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_CLEAN=True, distribution-scope=public, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T20:58:41.150 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:41 smithi187 podman[55570]: 2024-04-07 20:58:41.149368592 +0000 UTC m=+0.387385082 container init 821ef5ab9ce7386a3ede657222fd53e59e006c5ec22172b737a9053c33f0035b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5-deactivate, architecture=x86_64, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, ceph=True, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vendor=Red Hat, Inc., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-04-07T20:58:41.431 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:41 smithi187 podman[55570]: 2024-04-07 20:58:41.15251655 +0000 UTC m=+0.390533040 container start 821ef5ab9ce7386a3ede657222fd53e59e006c5ec22172b737a9053c33f0035b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, release=754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True) 2024-04-07T20:58:41.431 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:41 smithi187 podman[55570]: 2024-04-07 20:58:41.224424595 +0000 UTC m=+0.462441087 container attach 821ef5ab9ce7386a3ede657222fd53e59e006c5ec22172b737a9053c33f0035b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5-deactivate, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, version=8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, architecture=x86_64, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-07T20:58:41.806 INFO:journalctl@ceph.osd.5.smithi187.stdout:Apr 07 20:58:41 smithi187 podman[55570]: 2024-04-07 20:58:41.432320579 +0000 UTC m=+0.670337071 container died 821ef5ab9ce7386a3ede657222fd53e59e006c5ec22172b737a9053c33f0035b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-5-deactivate, release=754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, architecture=x86_64, maintainer=Guillaume Abrioux ) 2024-04-07T20:58:42.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:41 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:41.627Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:42.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:41 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:41.627Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:42.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:41 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:41.627Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:42.118 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:42.003+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:42.118 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:42.003+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:42.118 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:42.003+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:42.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-07T20:58:42.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:42.337 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-07T20:58:42.337 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-07T20:58:42.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.6 2024-04-07T20:58:42.379 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:42.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:42.380 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:42.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:42.380 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:42.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:42.807 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 systemd[1]: Stopping Ceph osd.6 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:42.807 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 bash[55688]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.6" found: no such container 2024-04-07T20:58:42.807 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:42.522+0000 7f417d89d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:42.807 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:42.522+0000 7f417d89d700 -1 osd.6 98 *** Got signal Terminated *** 2024-04-07T20:58:42.807 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:42 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:42.522+0000 7f417d89d700 -1 osd.6 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:43.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:43.117+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:43.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:43.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:43.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:43.118+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:43.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:43.023+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:43.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:43.023+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:43.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:43.023+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:43.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:43.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:44.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:43 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:43.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:44.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:43.997+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:44.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:43.997+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:44.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:43 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:43.997+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:44.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:44 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:44.126+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:44.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:44 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:44.126+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:44.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:44 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:44.126+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:45.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:45.146+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:45.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:45.146+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:45.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:45.146+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:45.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:45.038+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:45.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:45.038+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:45.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:45.038+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:45.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:45 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:45.038+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:46.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:46.162+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:46.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:46.162+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:46.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:46.162+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:46.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:46.031+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:46.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:46.031+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:46.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:46.031+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:46.308 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:46 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:46.031+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:47.557 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:47.169+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:55.050709+0000 front 2024-04-07T20:57:55.050643+0000 (oldest deadline 2024-04-07T20:58:21.451154+0000) 2024-04-07T20:58:47.557 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:47.169+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:04.852636+0000 front 2024-04-07T20:58:04.852621+0000 (oldest deadline 2024-04-07T20:58:27.152610+0000) 2024-04-07T20:58:47.557 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:47.169+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.253567+0000 front 2024-04-07T20:58:11.253623+0000 (oldest deadline 2024-04-07T20:58:35.953465+0000) 2024-04-07T20:58:47.557 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6[36294]: 2024-04-07T20:58:47.169+0000 7f4178234700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:21.655446+0000 front 2024-04-07T20:58:21.655471+0000 (oldest deadline 2024-04-07T20:58:46.355397+0000) 2024-04-07T20:58:47.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:47.063+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:47.558 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:47.063+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:47.558 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:47.063+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:47.558 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:47 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:47.063+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:48.307 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 podman[55699]: 2024-04-07 20:58:48.030287575 +0000 UTC m=+5.615395204 container died b45cbc92f59088f630635c55791448936247064cf47087abaf5fedab09173f15 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6, GIT_CLEAN=True, ceph=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, architecture=x86_64, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-type=git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-04-07T20:58:48.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:48 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:48.081+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:48.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:48 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:48.081+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:48.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:48 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:48.081+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:48.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:48 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:48.081+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:48.681 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 podman[55699]: 2024-04-07 20:58:48.405750618 +0000 UTC m=+5.990858256 container cleanup b45cbc92f59088f630635c55791448936247064cf47087abaf5fedab09173f15 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, distribution-scope=public, vcs-type=git, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, ceph=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=) 2024-04-07T20:58:48.681 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 bash[55699]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6 2024-04-07T20:58:48.681 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 bash[55734]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.6" found: no such container 2024-04-07T20:58:49.021 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 podman[55721]: 2024-04-07 20:58:48.681860242 +0000 UTC m=+0.639041140 container remove b45cbc92f59088f630635c55791448936247064cf47087abaf5fedab09173f15 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_BRANCH=HEAD) 2024-04-07T20:58:49.021 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 podman[55800]: 2024-04-07 20:58:48.809626868 +0000 UTC m=+0.020020082 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:49.021 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:48 smithi187 podman[55800]: 2024-04-07 20:58:48.916006422 +0000 UTC m=+0.126399635 container create dab8669d44b3014be771d84e69b737d290d373d31b56340d115e3967383e5781 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6-deactivate, io.buildah.version=1.19.8, vcs-type=git, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, name=centos-stream, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, version=8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vendor=Red Hat, Inc., architecture=x86_64, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-07T20:58:49.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:49.083+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:49.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:49.083+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:49.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:49.083+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:49.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:49 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:49.083+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:49.308 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:49 smithi187 podman[55800]: 2024-04-07 20:58:49.126789259 +0000 UTC m=+0.337182473 container init dab8669d44b3014be771d84e69b737d290d373d31b56340d115e3967383e5781 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6-deactivate, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc.) 2024-04-07T20:58:49.308 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:49 smithi187 podman[55800]: 2024-04-07 20:58:49.130030922 +0000 UTC m=+0.340424143 container start dab8669d44b3014be771d84e69b737d290d373d31b56340d115e3967383e5781 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, version=8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, distribution-scope=public, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-04-07T20:58:49.308 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:49 smithi187 podman[55800]: 2024-04-07 20:58:49.210094985 +0000 UTC m=+0.420488215 container attach dab8669d44b3014be771d84e69b737d290d373d31b56340d115e3967383e5781 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754) 2024-04-07T20:58:49.695 INFO:journalctl@ceph.osd.6.smithi187.stdout:Apr 07 20:58:49 smithi187 podman[55800]: 2024-04-07 20:58:49.4107349 +0000 UTC m=+0.621128109 container died dab8669d44b3014be771d84e69b737d290d373d31b56340d115e3967383e5781 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-6-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, maintainer=Guillaume Abrioux , architecture=x86_64, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.19.8, distribution-scope=public) 2024-04-07T20:58:49.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.6.service' 2024-04-07T20:58:50.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:50.056 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-07T20:58:50.056 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-07T20:58:50.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.7 2024-04-07T20:58:50.282 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.104+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:50.282 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.104+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:50.282 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.104+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:50.282 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.104+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:50.282 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 systemd[1]: Stopping Ceph osd.7 for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:50.283 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 bash[55880]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.7" found: no such container 2024-04-07T20:58:50.283 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.282+0000 7fae41402700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T20:58:50.283 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.282+0000 7fae41402700 -1 osd.7 98 *** Got signal Terminated *** 2024-04-07T20:58:50.283 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:50 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:50.282+0000 7fae41402700 -1 osd.7 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T20:58:51.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:51 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:51.138+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:51.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:51 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:51.138+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:51.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:51 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:51.138+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:51.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:51 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:51.138+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:52.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:51.627Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:52.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:51.627Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:52.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:51 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:51.627Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:52.556 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:52.119+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:52.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:52.119+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:52.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:52.119+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:52.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:52.119+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:52.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:52 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:52.119+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-04-07T20:58:29.735737+0000 front 2024-04-07T20:58:29.735816+0000 (oldest deadline 2024-04-07T20:58:52.035774+0000) 2024-04-07T20:58:53.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:53 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:53.071+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:53.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:53 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:53.071+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:53.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:53 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:53.071+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:53.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:53 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:53.071+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:53.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:53 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:53.071+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-04-07T20:58:29.735737+0000 front 2024-04-07T20:58:29.735816+0000 (oldest deadline 2024-04-07T20:58:52.035774+0000) 2024-04-07T20:58:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:58:53.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:53.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:58:54.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:58:53 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:58:53.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:58:54.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.032+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:54.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.032+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:54.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.032+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:54.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.032+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:54.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.032+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-04-07T20:58:29.735737+0000 front 2024-04-07T20:58:29.735816+0000 (oldest deadline 2024-04-07T20:58:52.035774+0000) 2024-04-07T20:58:55.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.999+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6806 osd.0 since back 2024-04-07T20:57:56.630886+0000 front 2024-04-07T20:57:56.630780+0000 (oldest deadline 2024-04-07T20:58:19.530900+0000) 2024-04-07T20:58:55.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.999+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6814 osd.1 since back 2024-04-07T20:58:03.631939+0000 front 2024-04-07T20:58:03.631953+0000 (oldest deadline 2024-04-07T20:58:25.931899+0000) 2024-04-07T20:58:55.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.999+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6822 osd.2 since back 2024-04-07T20:58:11.733253+0000 front 2024-04-07T20:58:11.733304+0000 (oldest deadline 2024-04-07T20:58:37.033205+0000) 2024-04-07T20:58:55.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.999+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6830 osd.3 since back 2024-04-07T20:58:20.334902+0000 front 2024-04-07T20:58:20.334873+0000 (oldest deadline 2024-04-07T20:58:45.034891+0000) 2024-04-07T20:58:55.307 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:54 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7[39258]: 2024-04-07T20:58:54.999+0000 7fae3ed9f700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-04-07T20:58:29.735737+0000 front 2024-04-07T20:58:29.735816+0000 (oldest deadline 2024-04-07T20:58:52.035774+0000) 2024-04-07T20:58:56.007 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:55 smithi187 podman[55891]: 2024-04-07 20:58:55.73516582 +0000 UTC m=+5.557012591 container died bf0628d563836c6d810422cc120f9d6dd225e2f3b620f208734d0a43a1c7b910 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-type=git, ceph=True, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64) 2024-04-07T20:58:56.305 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:56 smithi187 podman[55891]: 2024-04-07 20:58:56.198981567 +0000 UTC m=+6.020828339 container cleanup bf0628d563836c6d810422cc120f9d6dd225e2f3b620f208734d0a43a1c7b910 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, name=centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , version=8, distribution-scope=public, vendor=Red Hat, Inc.) 2024-04-07T20:58:56.306 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:56 smithi187 bash[55891]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7 2024-04-07T20:58:56.556 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:56 smithi187 bash[55948]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd.7" found: no such container 2024-04-07T20:58:57.421 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:57 smithi187 podman[55910]: 2024-04-07 20:58:57.087295179 +0000 UTC m=+1.348484341 container remove bf0628d563836c6d810422cc120f9d6dd225e2f3b620f208734d0a43a1c7b910 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , version=8, ceph=True, com.redhat.component=centos-stream-container) 2024-04-07T20:58:57.696 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:57 smithi187 podman[56056]: 2024-04-07 20:58:57.47797686 +0000 UTC m=+0.019735462 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T20:58:57.697 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:57 smithi187 podman[56056]: 2024-04-07 20:58:57.590252464 +0000 UTC m=+0.132011054 container create b02041effa277351d4ccec9558c43f990e7987cf3560acbd450642bffd751b7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, release=754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-type=git, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream) 2024-04-07T20:58:58.057 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:57 smithi187 podman[56056]: 2024-04-07 20:58:57.809529263 +0000 UTC m=+0.351287856 container init b02041effa277351d4ccec9558c43f990e7987cf3560acbd450642bffd751b7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, release=754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T20:58:58.057 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:57 smithi187 podman[56056]: 2024-04-07 20:58:57.812952345 +0000 UTC m=+0.354710935 container start b02041effa277351d4ccec9558c43f990e7987cf3560acbd450642bffd751b7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T20:58:58.057 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:57 smithi187 podman[56056]: 2024-04-07 20:58:57.892817815 +0000 UTC m=+0.434576408 container attach b02041effa277351d4ccec9558c43f990e7987cf3560acbd450642bffd751b7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7-deactivate, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, release=754, architecture=x86_64, GIT_BRANCH=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vcs-type=git, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , name=centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T20:58:58.386 INFO:journalctl@ceph.osd.7.smithi187.stdout:Apr 07 20:58:58 smithi187 podman[56056]: 2024-04-07 20:58:58.115025764 +0000 UTC m=+0.656784366 container died b02041effa277351d4ccec9558c43f990e7987cf3560acbd450642bffd751b7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-osd-7-deactivate, maintainer=Guillaume Abrioux , vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, version=8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T20:58:58.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@osd.7.service' 2024-04-07T20:58:58.742 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:58.743 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-07T20:58:58.743 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-07T20:58:58.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@prometheus.a 2024-04-07T20:58:58.973 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 systemd[1]: Stopping Ceph prometheus.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:58:58.973 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 bash[56228]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-07T20:58:59.256 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.973Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-07T20:58:59.257 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-07T20:58:59.258 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-07T20:58:59.258 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-07T20:58:59.258 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-07T20:58:59.258 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a[45630]: ts=2024-04-07T20:58:58.974Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-07T20:58:59.258 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:58 smithi187 podman[56247]: 2024-04-07 20:58:58.984280503 +0000 UTC m=+0.126103531 container died ac118aa18f679baa6a057190b30bb75e6b1059693017e3920ccbbce181204fe9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:58:59.556 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:59 smithi187 podman[56247]: 2024-04-07 20:58:59.356987494 +0000 UTC m=+0.498810539 container cleanup ac118aa18f679baa6a057190b30bb75e6b1059693017e3920ccbbce181204fe9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-07T20:58:59.557 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:59 smithi187 bash[56247]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-07T20:58:59.557 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Apr 07 20:58:59 smithi187 bash[56296]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-07T20:58:59.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@prometheus.a.service' 2024-04-07T20:58:59.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T20:58:59.658 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-07T20:58:59.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-07T20:59:01.628 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:01 smithi177 systemd[1]: Stopping Ceph node-exporter.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:59:01.628 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:01 smithi177 bash[65602]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter.a" found: no such container 2024-04-07T20:59:01.628 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:01 smithi177 podman[65615]: 2024-04-07 20:59:01.484231584 +0000 UTC m=+0.113353460 container died 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-07T20:59:01.888 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:01 smithi177 podman[65615]: 2024-04-07 20:59:01.849001615 +0000 UTC m=+0.478123499 container cleanup 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-07T20:59:01.888 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:01 smithi177 bash[65615]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a 2024-04-07T20:59:01.888 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:01 smithi177 bash[65642]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter.a" found: no such container 2024-04-07T20:59:01.888 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:01 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=error ts=2024-04-07T20:59:01.628Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:59:01.888 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:01 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:59:01.629Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443//api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-04-07T20:59:01.889 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:01 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=warn ts=2024-04-07T20:59:01.629Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.187:8443/api/prometheus_receiver\": dial tcp 172.21.15.187:8443: connect: connection refused" 2024-04-07T20:59:02.393 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:02 smithi177 podman[65630]: 2024-04-07 20:59:02.083578112 +0000 UTC m=+0.598709935 container remove 001a28e563be0423a00a7236bab978e0d572684db077bbcdba692902d769d058 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-07T20:59:02.394 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:02 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-07T20:59:02.394 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:02 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-07T20:59:02.394 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:02 smithi177 systemd[1]: Stopped Ceph node-exporter.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:59:02.394 INFO:journalctl@ceph.node-exporter.a.smithi177.stdout:Apr 07 20:59:02 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.a.service: Consumed 3.387s CPU time. 2024-04-07T20:59:02.394 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:02 smithi177 systemd[1]: Stopping Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:59:02.395 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:02 smithi177 bash[65696]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-07T20:59:02.395 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:02 smithi177 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a[51051]: level=info ts=2024-04-07T20:59:02.394Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-07T20:59:02.678 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:02 smithi177 podman[65707]: 2024-04-07 20:59:02.42604544 +0000 UTC m=+0.122414908 container died bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:59:03.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:02 smithi177 podman[65707]: 2024-04-07 20:59:02.861964032 +0000 UTC m=+0.558333499 container cleanup bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:59:03.009 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:02 smithi177 bash[65707]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-07T20:59:03.418 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:03 smithi177 podman[65719]: 2024-04-07 20:59:03.046610897 +0000 UTC m=+0.617555363 container remove bdca865a7e70341156d0943f20b041839885b8d16dfe0775878d2089c612927e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-07T20:59:03.419 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:03 smithi177 podman[65719]: 2024-04-07 20:59:03.114163902 +0000 UTC m=+0.685108362 volume remove a4270123e7b9eb673eb9f1250213b81c80c8239e4ce4a7bd8747aed662a86ea2 2024-04-07T20:59:03.419 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:03 smithi177 bash[65733]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-07T20:59:03.419 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:03 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-07T20:59:03.419 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:03 smithi177 systemd[1]: Stopped Ceph alertmanager.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:59:03.419 INFO:journalctl@ceph.alertmanager.a.smithi177.stdout:Apr 07 20:59:03 smithi177 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@alertmanager.a.service: Consumed 1.072s CPU time. 2024-04-07T20:59:37.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-07T20:59:39.434 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:39 smithi187 systemd[1]: Stopping Ceph node-exporter.b for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:59:39.434 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:39 smithi187 bash[57426]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter.b" found: no such container 2024-04-07T20:59:39.690 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:39 smithi187 podman[57439]: 2024-04-07 20:59:39.434938662 +0000 UTC m=+0.124919750 container died 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-07T20:59:40.056 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:39 smithi187 podman[57439]: 2024-04-07 20:59:39.799077403 +0000 UTC m=+0.489058502 container cleanup 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-07T20:59:40.057 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:39 smithi187 bash[57439]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b 2024-04-07T20:59:40.057 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:39 smithi187 bash[57465]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter.b" found: no such container 2024-04-07T20:59:40.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 systemd[1]: Stopping Ceph grafana.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4... 2024-04-07T20:59:40.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 bash[57519]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana.a" found: no such container 2024-04-07T20:59:40.444 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:40 smithi187 podman[57453]: 2024-04-07 20:59:40.117044775 +0000 UTC m=+0.681937022 container remove 908396a184b00d9b8a3b0069c31f15a6d26ac97f97b6d19a8f973de5a1f9595d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-07T20:59:40.445 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:40 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-07T20:59:40.445 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:40 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-07T20:59:40.445 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:40 smithi187 systemd[1]: Stopped Ceph node-exporter.b for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:59:40.445 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Apr 07 20:59:40 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@node-exporter.b.service: Consumed 3.475s CPU time. 2024-04-07T20:59:40.711 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a[44698]: t=2024-04-07T20:59:40+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-07T20:59:40.712 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 podman[57530]: 2024-04-07 20:59:40.455455853 +0000 UTC m=+0.122621931 container died 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, description=Ceph Grafana Container, io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , release=236.1648460182, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-07T20:59:40.986 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 podman[57530]: 2024-04-07 20:59:40.794817018 +0000 UTC m=+0.461983104 container cleanup 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a, vendor=Red Hat, Inc., architecture=x86_64, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.openshift.expose-services=, maintainer=Paul Cuzner , distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, name=ubi8, io.openshift.tags=base rhel8, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T20:59:40.987 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 bash[57530]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a 2024-04-07T20:59:41.249 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:40 smithi187 podman[57544]: 2024-04-07 20:59:40.987881032 +0000 UTC m=+0.532585099 container remove 7b41716cc5e1a8aeec86a9a9f1afcad8a252749284150c00771cb8261a17f6bf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana-a, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, release=236.1648460182, io.buildah.version=1.24.2, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base rhel8, io.openshift.expose-services=, vcs-type=git, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-07T20:59:41.249 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:41 smithi187 bash[57557]: Error: no container with name or ID "ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4-grafana.a" found: no such container 2024-04-07T20:59:41.249 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:41 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-04-07T20:59:41.249 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:41 smithi187 systemd[1]: Stopped Ceph grafana.a for c7f93d4c-f51f-11ee-b649-cb9ed24678a4. 2024-04-07T20:59:41.249 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Apr 07 20:59:41 smithi187 systemd[1]: ceph-c7f93d4c-f51f-11ee-b649-cb9ed24678a4@grafana.a.service: Consumed 3.045s CPU time. 2024-04-07T21:00:03.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T21:00:03.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T21:00:03.559 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-07T21:00:03.561 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/remote/smithi177/crash 2024-04-07T21:00:03.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/crash -- . 2024-04-07T21:00:03.674 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-07T21:00:03.675 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-04-07T21:00:03.677 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/remote/smithi187/crash 2024-04-07T21:00:03.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/crash -- . 2024-04-07T21:00:03.727 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-07T21:00:03.727 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2024-04-07T21:00:03.728 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-07T21:00:03.728 DEBUG:teuthology.orchestra.run.smithi177:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-07T21:00:03.760 INFO:teuthology.orchestra.run.smithi177.stdout:2024-04-07T20:47:45.900155+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-07T21:00:03.760 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-07T21:00:03.760 INFO:tasks.cephadm:Compressing logs... 2024-04-07T21:00:03.761 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-07T21:00:03.803 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-07T21:00:03.829 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-07T21:00:03.830 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-04-07T21:00:03.830 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.a.log 2024-04-07T21:00:03.830 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi177.soabjy.log 2024-04-07T21:00:03.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-07T21:00:03.841 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr: /var/log/ceph/cephadm.log 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.6.log 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi187.fluayj.log 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi187.fluayj.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.4.log 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr: 62.6% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi187.fluayj.log.gz 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.7.log 2024-04-07T21:00:03.842 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.audit.log: /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.5.log 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mgr.x.log 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 81.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr: 90.3% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi187.xszzor.log 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-07T21:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log: /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi187.xszzor.log: 62.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi187.xszzor.log.gz 2024-04-07T21:00:03.844 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.b.log 2024-04-07T21:00:03.844 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-volume.log: 87.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-07T21:00:03.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.a.log: /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi177.soabjy.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi177.pbcdof.log 2024-04-07T21:00:03.844 INFO:teuthology.orchestra.run.smithi177.stderr: 62.5% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi177.soabjy.log.gz 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.3.log 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi177.pbcdof.log: /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.1.log: 62.6% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi177.pbcdof.log.gz 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mgr.y.log 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/tcmu-runner.log 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-07T21:00:03.845 INFO:teuthology.orchestra.run.smithi177.stderr: 66.3% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/tcmu-runner.log.gz 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 89.3% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log: 93.9% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr: 92.8% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-07T21:00:03.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.c.log 2024-04-07T21:00:03.847 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.b.log: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-07T21:00:03.847 INFO:teuthology.orchestra.run.smithi187.stderr: 90.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mgr.x.log.gz 2024-04-07T21:00:03.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-volume.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-07T21:00:03.857 INFO:teuthology.orchestra.run.smithi187.stderr: 92.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-07T21:00:03.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.c.log: 92.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-07T21:00:03.976 INFO:teuthology.orchestra.run.smithi177.stderr: 87.5% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mgr.y.log.gz 2024-04-07T21:00:04.008 INFO:teuthology.orchestra.run.smithi187.stderr: 92.3% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.b.log.gz 2024-04-07T21:00:04.144 INFO:teuthology.orchestra.run.smithi177.stderr: 92.2% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.c.log.gz 2024-04-07T21:00:04.147 INFO:teuthology.orchestra.run.smithi187.stderr: 93.2% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-07T21:00:04.202 INFO:teuthology.orchestra.run.smithi177.stderr: 93.1% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-07T21:00:04.233 INFO:teuthology.orchestra.run.smithi187.stderr: 93.4% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-07T21:00:04.257 INFO:teuthology.orchestra.run.smithi177.stderr: 93.5% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-07T21:00:04.283 INFO:teuthology.orchestra.run.smithi187.stderr: 93.8% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-07T21:00:04.302 INFO:teuthology.orchestra.run.smithi177.stderr: 93.5% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-07T21:00:04.343 INFO:teuthology.orchestra.run.smithi187.stderr: 93.6% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-07T21:00:04.345 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-07T21:00:04.345 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m0.528s 2024-04-07T21:00:04.345 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m1.834s 2024-04-07T21:00:04.345 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m0.100s 2024-04-07T21:00:04.397 INFO:teuthology.orchestra.run.smithi177.stderr: 91.2% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-mon.a.log.gz 2024-04-07T21:00:04.413 INFO:teuthology.orchestra.run.smithi177.stderr: 93.5% -- replaced with /var/log/ceph/c7f93d4c-f51f-11ee-b649-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-07T21:00:04.414 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-07T21:00:04.415 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m0.599s 2024-04-07T21:00:04.415 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m2.811s 2024-04-07T21:00:04.415 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.122s 2024-04-07T21:00:04.415 INFO:tasks.cephadm:Archiving logs... 2024-04-07T21:00:04.415 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/remote/smithi177/log 2024-04-07T21:00:04.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-07T21:00:04.849 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/remote/smithi187/log 2024-04-07T21:00:04.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-07T21:00:05.152 INFO:tasks.cephadm:Removing cluster... 2024-04-07T21:00:05.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 --force 2024-04-07T21:00:05.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 --force 2024-04-07T21:00:05.712 INFO:tasks.cephadm:Removing cephadm ... 2024-04-07T21:00:05.712 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-07T21:00:05.730 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-07T21:00:05.746 INFO:tasks.cephadm:Teardown complete 2024-04-07T21:00:05.747 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-07T21:00:05.757 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-07T21:00:05.757 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T21:00:05.772 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T21:00:05.784 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-07T21:00:05.788 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T21:00:05.788 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-07T21:00:05.788 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -162us[ -182us] +/- 82ms 2024-04-07T21:00:05.788 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -696us[ -716us] +/- 69ms 2024-04-07T21:00:05.788 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +857us[ +857us] +/- 70ms 2024-04-07T21:00:05.788 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T21:00:05.799 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-07T21:00:05.803 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T21:00:05.803 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-07T21:00:05.803 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -166us[ -166us] +/- 82ms 2024-04-07T21:00:05.803 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -688us[ -701us] +/- 69ms 2024-04-07T21:00:05.804 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +853us[ +840us] +/- 70ms 2024-04-07T21:00:05.804 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T21:00:05.804 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-07T21:00:05.813 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-07T21:00:05.814 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-07T21:00:05.833 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-07T21:00:05.907 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-07T21:00:05.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-07T21:00:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T21:00:06.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-07T21:00:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T21:00:06.037 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-07T21:00:06.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-07T21:00:06.074 INFO:teuthology.task.internal:Duration was 1155.224393 seconds 2024-04-07T21:00:06.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-07T21:00:06.092 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-07T21:00:06.092 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-07T21:00:06.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-07T21:00:06.135 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T21:00:06.138 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T21:00:06.542 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-07T21:00:06.542 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-07T21:00:06.543 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-07T21:00:06.603 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-04-07T21:00:06.603 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-07T21:00:06.625 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-07T21:00:06.625 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-07T21:00:06.646 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-07T21:00:06.693 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-07T21:00:06.693 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-07T21:00:06.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-07T21:00:07.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-07T21:00:07.038 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-07T21:00:07.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-07T21:00:07.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-07T21:00:07.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-07T21:00:07.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-07T21:00:07.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-07T21:00:07.130 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-07T21:00:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-04-07T21:00:07.184 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-07T21:00:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T21:00:07.207 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-07T21:00:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T21:00:07.239 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-07T21:00:07.248 INFO:teuthology.task.internal:Transferring archived files... 2024-04-07T21:00:07.249 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/remote/smithi177 2024-04-07T21:00:07.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-07T21:00:07.297 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282/remote/smithi187 2024-04-07T21:00:07.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-07T21:00:07.337 INFO:teuthology.task.internal:Removing archive directory... 2024-04-07T21:00:07.337 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-07T21:00:07.339 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-07T21:00:07.385 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-07T21:00:07.397 INFO:teuthology.task.internal:Not uploading archives. 2024-04-07T21:00:07.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-07T21:00:07.409 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-07T21:00:07.410 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-07T21:00:07.412 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-07T21:00:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: 266034 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 7 21:00 /home/ubuntu/cephtest 2024-04-07T21:00:07.439 INFO:teuthology.orchestra.run.smithi187.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 7 21:00 /home/ubuntu/cephtest 2024-04-07T21:00:07.440 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-07T21:00:07.451 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-07T21:00:07.472 INFO:teuthology.nuke:Checking targets against current locks 2024-04-07T21:00:07.503 DEBUG:teuthology.nuke:shortname: smithi177 2024-04-07T21:00:07.503 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-07T21:00:07.526 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 20:31:25.337603', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOjRNcyFziMat0n9inS6w2GDfnJ+8G/WWtvjs0oGMcrS9h0s70dVGCJNHk03Ic8zck+Fw84cKvzp/wH6a7+vsho='} 2024-04-07T21:00:07.531 DEBUG:teuthology.nuke:shortname: smithi187 2024-04-07T21:00:07.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-07T21:00:07.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634282', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 20:31:25.339298', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXVEGOu3DIVpWJjYDDZ5/MvQK9d7cePXfofX/S+0aDruk/NFwKjz8c1ZqolJ+b1FNWD/zbXMJZHGi04ddsgLeo='} 2024-04-07T21:00:07.582 INFO:teuthology.orchestra.console.smithi177:Power off 2024-04-07T21:00:07.583 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-07T21:00:07.608 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-04-07T21:00:07.609 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T21:00:07.624 INFO:teuthology.orchestra.console.smithi187:Power off 2024-04-07T21:00:07.624 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-07T21:00:07.634 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-07T21:00:07.650 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2024-04-07T21:00:07.650 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T21:00:07.675 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-07T21:00:11.635 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T21:00:11.676 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T21:00:11.746 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-07T21:00:11.786 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-07T21:00:15.750 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T21:00:15.787 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T21:00:15.861 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-04-07T21:00:15.898 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2024-04-07T21:00:15.963 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-04-07T21:00:15.999 INFO:teuthology.orchestra.console.smithi187:Power off completed 2024-04-07T21:00:16.105 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} duration: 1155.2243931293488 failure_reason: 'Command failed on smithi177 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7f93d4c-f51f-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c ''ceph orch ps''"' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a70562b49cb74f9ba9294ced52b5216b status: fail success: false 2024-04-07T21:00:16.105 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-07T21:00:16.197 INFO:teuthology.run:FAIL