2024-09-20T18:41:08.381 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T18:41:08.403 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T18:41:08.490 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726 branch: wip-guits-main-2024-09-20-1028 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} email: gabrioux@ibm.com first_in_suite: false job_id: '7912726' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-20-1028 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 workunit: branch: wip-guits-main-2024-09-20-1028 sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 9945 sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-20-1028 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3ck6jN31CVhSrQ1rv7NnEZUstzan5G29az/TO/hS4+jjImp2TH33VVvc4kpNg/Z954fkVH431pfYfd41AHCPs= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF1i4XZRbfI3OBDuI3rKm462+4kP29sF2nwOcKAX0hxZoWJELVEXUklgV2FSTbKnOd06rKV74GVPn0lLfeEUXCI= tasks: - cephadm: compiled_cephadm_branch: reef image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_12:24:58 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-20T18:41:08.490 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa; will attempt to use it 2024-09-20T18:41:08.491 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks 2024-09-20T18:41:08.491 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T18:41:08.492 INFO:teuthology.task.internal:Checking packages... 2024-09-20T18:41:08.512 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '88deadb8082f4a33832e228ccefc59cedba158a5' 2024-09-20T18:41:08.512 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T18:41:08.512 INFO:teuthology.packaging:ref: None 2024-09-20T18:41:08.513 INFO:teuthology.packaging:tag: None 2024-09-20T18:41:08.513 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-20-1028 2024-09-20T18:41:08.513 INFO:teuthology.packaging:sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T18:41:08.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-20-1028 2024-09-20T18:41:08.653 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5099.g88deadb8 2024-09-20T18:41:08.655 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T18:41:08.662 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T18:41:08.662 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T18:41:08.721 INFO:teuthology.task.internal:Saving configuration 2024-09-20T18:41:08.733 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T18:41:08.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T18:41:08.780 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 18:35:37.849358', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3ck6jN31CVhSrQ1rv7NnEZUstzan5G29az/TO/hS4+jjImp2TH33VVvc4kpNg/Z954fkVH431pfYfd41AHCPs='} 2024-09-20T18:41:08.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 18:35:37.847423', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF1i4XZRbfI3OBDuI3rKm462+4kP29sF2nwOcKAX0hxZoWJELVEXUklgV2FSTbKnOd06rKV74GVPn0lLfeEUXCI='} 2024-09-20T18:41:08.804 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T18:41:08.810 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-20T18:41:08.810 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-20T18:41:08.810 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T18:41:08.901 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd3cae0c280>, signals=[15]) 2024-09-20T18:41:08.901 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T18:41:08.907 INFO:teuthology.task.internal:Opening connections... 2024-09-20T18:41:08.907 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-09-20T18:41:08.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:41:08.984 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-09-20T18:41:08.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:41:09.058 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T18:41:09.064 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-09-20T18:41:09.083 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-09-20T18:41:09.083 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-09-20T18:41:09.140 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T18:41:09.141 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T18:41:09.141 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-09-20T18:41:09.160 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-09-20T18:41:09.178 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-09-20T18:41:09.178 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-09-20T18:41:09.236 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-09-20T18:41:09.236 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-09-20T18:41:09.236 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-09-20T18:41:09.236 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-09-20T18:41:09.236 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T18:41:09.237 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T18:41:09.237 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-09-20T18:41:09.268 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T18:41:09.282 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T18:41:09.355 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T18:41:09.355 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-09-20T18:41:09.357 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-09-20T18:41:09.374 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T18:41:09.397 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T18:41:09.397 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-09-20T18:41:09.415 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-09-20T18:41:09.437 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T18:41:09.491 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T18:41:09.506 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T18:41:09.507 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T18:41:09.507 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T18:41:09.507 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T18:41:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T18:41:09.524 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-20T18:41:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T18:41:09.525 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-20T18:41:09.581 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-503.el9.x86_64 2024-09-20T18:41:09.582 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-503.el9.x86_64 2024-09-20T18:41:09.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2024-09-20T18:41:09.583 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-503.el9.x86_64 2024-09-20T18:41:09.583 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-503.el9.x86_64 2024-09-20T18:41:09.584 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-09-20T18:41:11.965 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-20T18:41:13.574 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 37 MB/s | 20 MB 00:00 2024-09-20T18:41:15.968 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.3 MB 00:04 2024-09-20T18:41:18.983 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-20T18:41:19.144 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 20 MB 00:02 2024-09-20T18:41:20.918 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-20T18:41:21.963 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-20T18:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-20T18:41:25.530 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 19 kB 00:00 2024-09-20T18:41:26.658 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-20T18:41:27.460 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-20T18:41:29.375 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T18:41:29.375 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T18:41:29.471 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-20T18:41:29.478 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-20T18:41:29.478 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-20T18:41:29.478 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-20T18:41:29.478 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-20T18:41:29.478 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T18:41:29.478 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-20T18:41:29.479 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-09-20T18:41:29.482 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:41:29.483 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 79 M 2024-09-20T18:41:29.483 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 126 M 2024-09-20T18:41:29.483 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-20T18:41:30.336 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 63 kB/s | 36 kB 00:00 2024-09-20T18:41:31.295 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-20T18:41:31.970 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-20T18:41:32.263 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-20T18:41:32.654 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-20T18:41:32.654 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-20T18:41:32.654 INFO:teuthology.orchestra.run.smithi194.stdout:Total 25 MB/s | 79 MB 00:03 2024-09-20T18:41:33.341 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-20T18:41:33.692 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-20T18:41:33.692 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-20T18:41:34.178 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T18:41:34.178 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T18:41:34.269 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:41:34.277 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-20T18:41:34.278 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-20T18:41:34.278 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-09-20T18:41:34.278 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:41:34.278 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 79 M 2024-09-20T18:41:34.278 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 126 M 2024-09-20T18:41:34.278 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-20T18:41:34.490 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 317 kB/s | 36 kB 00:00 2024-09-20T18:41:34.640 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-20T18:41:34.640 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-20T18:41:35.090 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 48 MB/s | 29 MB 00:00 2024-09-20T18:41:35.557 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-20T18:41:35.949 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 22 MB/s | 35 MB 00:01 2024-09-20T18:41:35.949 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-20T18:41:35.949 INFO:teuthology.orchestra.run.smithi076.stdout:Total 47 MB/s | 79 MB 00:01 2024-09-20T18:41:36.629 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-20T18:41:37.000 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-20T18:41:37.001 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-20T18:41:37.085 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-20T18:41:37.757 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T18:41:37.863 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T18:41:37.925 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-20T18:41:37.925 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-20T18:41:40.357 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-20T18:41:40.648 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T18:41:40.781 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T18:41:41.000 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T18:41:41.081 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T18:41:43.821 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T18:41:43.956 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T18:41:45.583 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T18:41:45.631 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:41:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T18:41:48.915 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:41:50.492 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:41:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:28.721 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:28.760 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:29.123 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:29.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T18:42:29.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T18:42:29.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:42:30.003 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-20T18:42:30.337 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T18:42:30.417 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T18:42:30.417 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T18:42:30.417 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T18:42:30.417 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T18:42:30.417 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T18:42:30.417 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-20T18:42:30.418 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T18:42:30.418 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T18:42:30.477 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T18:42:30.477 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T18:42:30.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 82.el9 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-09-20T18:42:30.517 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8274670 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-09-20T18:42:30.518 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:42:30.519 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T18:42:30.519 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T18:42:30.519 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T18:42:30.519 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-09-20T18:42:30.519 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:42:30.519 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T18:42:30.519 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T18:42:30.519 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-508.el9.x86_64 2024-09-20T18:42:30.519 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T18:42:31.142 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-09-20T18:42:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:32.493 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T18:42:32.521 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-09-20T18:42:32.523 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-09-20T18:42:32.539 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.FRbyzmM7gr 2024-09-20T18:42:32.539 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FRbyzmM7gr 2024-09-20T18:42:32.606 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.FRbyzmM7gr 2024-09-20T18:42:32.647 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:32.647 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T18:42:32.647 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T18:42:32.647 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T18:42:32.741 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.FRbyzmM7gr is 6KB 2024-09-20T18:42:32.789 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.FRbyzmM7gr 2024-09-20T18:42:32.804 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T18:42:32.866 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T18:42:32.866 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T18:42:32.866 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T18:42:32.866 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T18:42:32.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:42:33.481 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-20T18:42:33.846 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T18:42:33.933 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.933 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T18:42:33.933 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T18:42:33.933 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T18:42:33.933 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T18:42:33.933 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-20T18:42:33.933 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T18:42:33.933 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T18:42:33.992 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T18:42:33.993 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T18:42:33.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-09-20T18:42:34.030 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-09-20T18:42:34.030 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-09-20T18:42:34.030 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-09-20T18:42:34.030 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 82.el9 2024-09-20T18:42:34.030 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8274670 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:42:34.031 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T18:42:34.032 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T18:42:34.032 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T18:42:34.032 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-09-20T18:42:34.032 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:42:34.032 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T18:42:34.033 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T18:42:34.033 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-508.el9.x86_64 2024-09-20T18:42:34.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T18:42:34.699 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-09-20T18:42:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T18:42:36.043 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-09-20T18:42:36.045 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-20T18:42:36.060 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.lcSoeVY7qC 2024-09-20T18:42:36.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lcSoeVY7qC 2024-09-20T18:42:36.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.lcSoeVY7qC 2024-09-20T18:42:36.274 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.lcSoeVY7qC is 6KB 2024-09-20T18:42:36.294 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.lcSoeVY7qC 2024-09-20T18:42:36.309 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T18:42:36.372 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T18:42:36.372 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T18:42:36.372 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T18:42:36.372 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T18:42:36.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T18:42:36.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-09-20T18:43:03.051 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-20T18:43:03.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-20T18:43:03.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:43:06.542 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-20T18:43:06.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:43:06.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:43:21.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-09-20T18:43:25.039 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-20T18:43:30.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-20T18:43:30.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:43:33.523 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-09-20T18:43:34.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:43:34.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:43:37.103 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-20T18:43:45.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-20T18:43:45.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:43:49.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:43:49.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:43:52.175 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-20T18:44:07.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:44:07.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:44:17.331 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-20T18:44:17.648 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-20T18:44:17.648 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T18:44:17.649 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-20T18:44:17.663 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-508.el9.x86_64 2024-09-20T18:44:17.664 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T18:44:17.664 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T18:44:17.664 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T18:44:18.664 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-20T18:44:18.665 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T18:44:18.744 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-09-20T18:44:18.762 DEBUG:teuthology.parallel:result is None 2024-09-20T18:44:25.711 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-20T18:44:43.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:44:43.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:44:44.023 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-20T18:44:44.413 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:44:44.413 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T18:44:44.414 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-20T18:44:44.430 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-508.el9.x86_64 2024-09-20T18:44:44.430 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T18:44:44.430 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T18:44:44.430 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T18:44:45.431 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-20T18:44:45.431 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T18:44:45.523 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2024-09-20T18:44:45.544 DEBUG:teuthology.parallel:result is None 2024-09-20T18:44:45.544 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T18:44:45.549 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T18:44:45.550 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T18:44:45.552 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T18:44:45.572 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T18:44:45.613 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T18:44:45.665 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T18:44:45.665 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T18:44:45.667 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T18:44:45.705 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T18:44:45.716 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T18:44:45.716 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T18:44:45.733 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T18:44:45.765 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:44:45.777 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:44:45.781 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:44:45.792 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:44:45.794 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T18:44:45.800 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T18:44:45.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T18:44:45.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T18:44:45.861 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T18:44:45.875 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T18:44:45.875 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T18:44:45.892 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T18:44:45.918 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T18:44:45.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T18:44:46.048 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T18:44:46.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T18:44:46.174 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:44:46.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T18:44:46.240 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T18:44:46.280 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T18:44:46.342 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T18:44:46.426 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T18:44:46.470 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:44:46.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T18:44:46.537 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-09-20T18:44:46.539 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-09-20T18:44:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T18:44:46.619 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T18:44:47.132 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T18:44:47.142 INFO:teuthology.task.internal:Starting timer... 2024-09-20T18:44:47.143 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T18:44:47.177 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T18:44:47.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-09-20T18:44:47.280 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-09-20T18:44:47.282 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-09-20T18:44:47.339 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-09-20T18:44:47.340 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T18:44:47.341 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-09-20T18:44:47.368 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-09-20T18:44:47.368 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-09-20T18:44:47.395 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-09-20T18:44:47.395 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-09-20T18:44:47.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T18:44:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T18:44:47.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T18:44:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T18:44:47.468 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T18:44:47.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-09-20T18:44:47.520 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-09-20T18:44:47.555 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T18:44:47.564 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T18:44:47.574 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T18:44:47.575 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T18:48:31.457 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-09-20T18:48:31.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:48:31.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:48:31.533 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-20T18:48:31.611 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-20T18:48:31.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-20T18:48:31.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:48:31.689 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-20T18:48:31.765 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-20T18:48:31.765 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T18:48:31.775 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T18:48:31.775 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T18:48:31.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T18:48:31.778 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T18:48:31.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T18:48:31.806 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T18:48:31.820 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T18:48:31.835 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T18:48:31.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-09-20T18:48:31.849 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T18:48:31.855 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-09-20T18:48:31.869 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T18:48:31.873 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-09-20T18:48:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T18:48:31.887 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-09-20T18:48:31.899 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T18:48:31.913 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T18:48:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-09-20T18:48:31.940 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T18:48:31.941 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-09-20T18:48:31.941 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.941 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.941 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.941 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.971 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-09-20T18:48:31.975 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T18:48:31.975 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-09-20T18:48:31.975 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.975 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.975 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.976 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T18:48:31.976 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T18:48:32.072 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'} 2024-09-20T18:48:32.073 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T18:48:32.073 INFO:tasks.cephadm:Cluster fsid is eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:48:32.073 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T18:48:32.073 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.76', 'mon.c': '[v2:172.21.15.76:3301,v1:172.21.15.76:6790]', 'mon.b': '172.21.15.194'} 2024-09-20T18:48:32.073 INFO:tasks.cephadm:First mon is mon.a on smithi076 2024-09-20T18:48:32.074 INFO:tasks.cephadm:First mgr is y 2024-09-20T18:48:32.074 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T18:48:32.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2024-09-20T18:48:32.100 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-09-20T18:48:32.125 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-20T18:48:32.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T18:48:32.268 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83186/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5099-g88deadb8', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5099.g88deadb8'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 12:11:57.149019', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-20-1028', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-20T18:48:32.510 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-20T18:48:32.512 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm 2024-09-20T18:48:32.512 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm 2024-09-20T18:48:32.512 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T18:48:33.010 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-r--r--. 1 ubuntu ubuntu 217226 Sep 20 18:48 /home/ubuntu/cephtest/cephadm 2024-09-20T18:48:33.011 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T18:48:33.514 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 217226 Sep 20 18:48 /home/ubuntu/cephtest/cephadm 2024-09-20T18:48:33.515 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T18:48:33.534 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T18:48:33.569 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-20T18:48:33.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-20T18:48:33.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-20T18:48:35.075 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T18:48:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T18:49:01.803 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-20T18:49:01.804 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)", 2024-09-20T18:49:01.804 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd", 2024-09-20T18:49:01.804 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-09-20T18:49:01.804 INFO:teuthology.orchestra.run.smithi194.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce" 2024-09-20T18:49:01.804 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-20T18:49:01.804 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)", 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd", 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce" 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-20T18:49:02.267 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-20T18:49:02.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2024-09-20T18:49:02.330 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-09-20T18:49:02.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2024-09-20T18:49:02.396 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-09-20T18:49:02.424 INFO:tasks.cephadm:Writing seed config... 2024-09-20T18:49:02.425 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-20T18:49:02.425 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T18:49:02.425 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T18:49:02.426 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-09-20T18:49:02.426 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T18:49:02.426 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T18:49:02.426 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T18:49:02.426 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T18:49:02.426 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T18:49:02.426 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:49:02.427 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T18:49:02.452 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eefb3510-7780-11ef-baf5-efdc52797490 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T18:49:02.452 DEBUG:teuthology.orchestra.run.smithi076:mon.a> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a.service 2024-09-20T18:49:02.495 DEBUG:teuthology.orchestra.run.smithi076:mgr.y> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y.service 2024-09-20T18:49:02.537 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T18:49:02.537 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid eefb3510-7780-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.76 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T18:49:02.690 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-20T18:49:02.690 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', 'eefb3510-7780-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.76', '--skip-admin-label'] 2024-09-20T18:49:02.722 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 5.2.2 2024-09-20T18:49:02.722 INFO:teuthology.orchestra.run.smithi076.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-20T18:49:02.723 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying podman|docker is present... 2024-09-20T18:49:02.745 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 5.2.2 2024-09-20T18:49:02.745 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying lvm2 is present... 2024-09-20T18:49:02.745 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying time synchronization is in place... 2024-09-20T18:49:02.752 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T18:49:02.752 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T18:49:02.758 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T18:49:02.758 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-09-20T18:49:02.765 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-09-20T18:49:02.771 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-09-20T18:49:02.771 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-09-20T18:49:02.771 INFO:teuthology.orchestra.run.smithi076.stdout:Repeating the final host check... 2024-09-20T18:49:02.793 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 5.2.2 2024-09-20T18:49:02.793 INFO:teuthology.orchestra.run.smithi076.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-20T18:49:02.793 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl is present 2024-09-20T18:49:02.793 INFO:teuthology.orchestra.run.smithi076.stdout:lvcreate is present 2024-09-20T18:49:02.800 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T18:49:02.800 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T18:49:02.806 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T18:49:02.806 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-09-20T18:49:02.812 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-09-20T18:49:02.819 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-09-20T18:49:02.819 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-09-20T18:49:02.819 INFO:teuthology.orchestra.run.smithi076.stdout:Host looks OK 2024-09-20T18:49:02.819 INFO:teuthology.orchestra.run.smithi076.stdout:Cluster fsid: eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:02.820 INFO:teuthology.orchestra.run.smithi076.stdout:Acquiring lock 140221318082720 on /run/cephadm/eefb3510-7780-11ef-baf5-efdc52797490.lock 2024-09-20T18:49:02.820 INFO:teuthology.orchestra.run.smithi076.stdout:Lock 140221318082720 acquired on /run/cephadm/eefb3510-7780-11ef-baf5-efdc52797490.lock 2024-09-20T18:49:02.820 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 3300 ... 2024-09-20T18:49:02.820 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 6789 ... 2024-09-20T18:49:02.820 INFO:teuthology.orchestra.run.smithi076.stdout:Base mon IP(s) is [172.21.15.76:3300, 172.21.15.76:6789], mon addrv is [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-09-20T18:49:02.823 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.76 metric 100 2024-09-20T18:49:02.823 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.76 metric 100 2024-09-20T18:49:02.825 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T18:49:02.825 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T18:49:02.826 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T18:49:02.828 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T18:49:02.828 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-20T18:49:02.828 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T18:49:02.828 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T18:49:02.828 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link noprefixroute 2024-09-20T18:49:02.828 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T18:49:02.829 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-09-20T18:49:02.829 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-09-20T18:49:02.829 INFO:teuthology.orchestra.run.smithi076.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-20T18:49:02.829 INFO:teuthology.orchestra.run.smithi076.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T18:49:02.830 INFO:teuthology.orchestra.run.smithi076.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Getting image source signatures 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:1f571018c56d8512357839d2ed445827ec316c3aed0fbfc68aff9046d4aceb99 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying config sha256:4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd 2024-09-20T18:49:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-20T18:49:04.174 INFO:teuthology.orchestra.run.smithi076.stdout:ceph: stdout ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable) 2024-09-20T18:49:04.174 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph version: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable) 2024-09-20T18:49:04.174 INFO:teuthology.orchestra.run.smithi076.stdout:Extracting ceph user uid/gid from container image... 2024-09-20T18:49:05.029 INFO:teuthology.orchestra.run.smithi076.stdout:stat: stdout 167 167 2024-09-20T18:49:05.029 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial keys... 2024-09-20T18:49:06.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQChw+1mtGyiFBAArCCDMlsgH35HD4xbzuqoZg== 2024-09-20T18:49:07.133 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQCiw+1mE3+UGBAAb5DmHABXA3s+N+0c+thIfg== 2024-09-20T18:49:07.997 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQCjw+1m+MnXHBAAMB6aEdiBfGEUT3GvC7gnlQ== 2024-09-20T18:49:07.998 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial monmap... 2024-09-20T18:49:08.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T18:49:08.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-20T18:49:08.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:08.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T18:49:08.887 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool for a [v2:172.21.15.76:3300,v1:172.21.15.76:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T18:49:08.888 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = pacific 2024-09-20T18:49:08.888 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: set fsid to eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:08.888 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T18:49:08.888 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:49:08.888 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mon... 2024-09-20T18:49:16.346 INFO:teuthology.orchestra.run.smithi076.stdout:create mon.a on 2024-09-20T18:49:17.104 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T18:49:17.730 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eefb3510-7780-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-eefb3510-7780-11ef-baf5-efdc52797490.target. 2024-09-20T18:49:17.730 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eefb3510-7780-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-eefb3510-7780-11ef-baf5-efdc52797490.target. 2024-09-20T18:49:18.144 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a 2024-09-20T18:49:18.144 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-20T18:49:18.307 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eefb3510-7780-11ef-baf5-efdc52797490.target.wants/ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-eefb3510-7780-11ef-baf5-efdc52797490@.service. 2024-09-20T18:49:18.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:18 smithi076 podman[23263]: 2024-09-20 18:49:18.712492242 +0000 UTC m=+0.296805657 container start eb76d104d8ee2e5ede667737bb6c0a19a8e66aba5385163b27701e3cdaa67a9d (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T18:49:18.818 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T18:49:18.818 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-09-20T18:49:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T18:49:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-09-20T18:49:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-09-20T18:49:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T18:49:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon to start... 2024-09-20T18:49:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon... 2024-09-20T18:49:19.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:18 smithi076 ceph-mon[23282]: mkfs eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:19.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:18 smithi076 ceph-mon[23282]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T18:49:19.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:18 smithi076 bash[23263]: eb76d104d8ee2e5ede667737bb6c0a19a8e66aba5385163b27701e3cdaa67a9d 2024-09-20T18:49:19.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:18 smithi076 systemd[1]: Started Ceph mon.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:49:19.865 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:19 smithi076 ceph-mon[23282]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T18:49:19.866 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:19 smithi076 ceph-mon[23282]: monmap e1: 1 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:49:19.866 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:19 smithi076 ceph-mon[23282]: fsmap 2024-09-20T18:49:19.866 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:19 smithi076 ceph-mon[23282]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T18:49:19.866 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:19 smithi076 ceph-mon[23282]: mgrmap e1: no daemons active 2024-09-20T18:49:19.866 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:19 smithi076 ceph-mon[23282]: from='client.? 172.21.15.76:0/3197780192' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T18:49:19.869 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout cluster: 2024-09-20T18:49:19.869 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout id: eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:19.869 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-20T18:49:19.869 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout services: 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.555816s) 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout data: 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pgs: 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:mon is available 2024-09-20T18:49:19.870 INFO:teuthology.orchestra.run.smithi076.stdout:Assimilating anything we can from ceph.conf... 2024-09-20T18:49:20.875 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:20.875 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T18:49:20.875 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:20.875 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T18:49:20.876 INFO:teuthology.orchestra.run.smithi076.stdout:Generating new minimal ceph.conf... 2024-09-20T18:49:20.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:20 smithi076 ceph-mon[23282]: from='client.? 172.21.15.76:0/2129834078' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T18:49:20.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:20 smithi076 ceph-mon[23282]: from='client.? 172.21.15.76:0/2129834078' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T18:49:21.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:21 smithi076 ceph-mon[23282]: from='client.? 172.21.15.76:0/1164836017' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:49:21.897 INFO:teuthology.orchestra.run.smithi076.stdout:Restarting the monitor... 2024-09-20T18:49:22.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:21 smithi076 systemd[1]: Stopping Ceph mon.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:49:22.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:22 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a[23278]: 2024-09-20T18:49:22.156+0000 7fbc0efaa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T18:49:22.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:22 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a[23278]: 2024-09-20T18:49:22.156+0000 7fbc0efaa640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T18:49:22.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:22 smithi076 podman[23522]: 2024-09-20 18:49:22.269255769 +0000 UTC m=+0.224616616 container died eb76d104d8ee2e5ede667737bb6c0a19a8e66aba5385163b27701e3cdaa67a9d (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T18:49:22.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:22 smithi076 podman[23522]: 2024-09-20 18:49:22.5981934 +0000 UTC m=+0.553554255 container cleanup eb76d104d8ee2e5ede667737bb6c0a19a8e66aba5385163b27701e3cdaa67a9d (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T18:49:22.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:22 smithi076 bash[23522]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a 2024-09-20T18:49:23.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:22 smithi076 podman[23535]: 2024-09-20 18:49:22.782295245 +0000 UTC m=+0.508708340 container remove eb76d104d8ee2e5ede667737bb6c0a19a8e66aba5385163b27701e3cdaa67a9d (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T18:49:23.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:23 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T18:49:23.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:23 smithi076 systemd[1]: Stopped Ceph mon.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:49:23.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:23 smithi076 systemd[1]: Starting Ceph mon.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:49:23.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:23 smithi076 podman[23629]: 2024-09-20 18:49:23.1264365 +0000 UTC m=+0.015707973 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T18:49:24.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:24 smithi076 podman[23629]: 2024-09-20 18:49:24.240721778 +0000 UTC m=+1.129993272 container create 856419dcdf1284f6960ad3b9b84af688230c38c666727a5c3187907570638b00 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T18:49:25.271 INFO:teuthology.orchestra.run.smithi076.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-20T18:49:25.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 podman[23629]: 2024-09-20 18:49:25.175676883 +0000 UTC m=+2.064948347 container init 856419dcdf1284f6960ad3b9b84af688230c38c666727a5c3187907570638b00 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T18:49:25.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 podman[23629]: 2024-09-20 18:49:25.179108551 +0000 UTC m=+2.068380003 container start 856419dcdf1284f6960ad3b9b84af688230c38c666727a5c3187907570638b00 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T18:49:25.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T18:49:25.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T18:49:25.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: pidfile_write: ignore empty --pid-file 2024-09-20T18:49:25.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: load: jerasure load: lrc 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: RocksDB version: 7.9.2 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Git sha 0 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: DB SUMMARY 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: DB Session ID: 8AA97AAX4YC40RPCVMTG 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: CURRENT file: CURRENT 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 83648 ; 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.error_if_exists: 0 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.create_if_missing: 0 2024-09-20T18:49:25.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.paranoid_checks: 1 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.env: 0x55b47dc7dbc0 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.info_log: 0x55b47ef0ebe0 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.statistics: (nil) 2024-09-20T18:49:25.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.use_fsync: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_log_file_size: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.allow_fallocate: 1 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.use_direct_reads: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T18:49:25.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.db_log_dir: 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.wal_dir: 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.write_buffer_manager: 0x55b47ef1c320 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T18:49:25.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.unordered_write: 0 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.row_cache: None 2024-09-20T18:49:25.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.wal_filter: None 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.two_write_queues: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.wal_compression: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.atomic_flush: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.log_readahead_size: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T18:49:25.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_background_jobs: 2 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_background_compactions: -1 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_subcompactions: 1 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T18:49:25.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_open_files: -1 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_background_flushes: -1 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Compression algorithms supported: 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kZSTD supported: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kXpressCompression supported: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kBZip2Compression supported: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T18:49:25.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kLZ4Compression supported: 1 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kZlibCompression supported: 1 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: kSnappyCompression supported: 1 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.merge_operator: 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_filter: None 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T18:49:25.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b47ef0e7e0) 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks: 1 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_top_level_index_and_filter: 1 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_type: 0 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_index_type: 0 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_shortening: 1 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: checksum: 4 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: no_block_cache: 0 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache: 0x55b47ef03350 2024-09-20T18:49:25.404 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_name: BinnedLRUCache 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_options: 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: capacity : 536870912 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_shard_bits : 4 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: strict_capacity_limit : 0 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: high_pri_pool_ratio: 0.000 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_compressed: (nil) 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: persistent_cache: (nil) 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size: 4096 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size_deviation: 10 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_restart_interval: 16 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_block_restart_interval: 1 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: metadata_block_size: 4096 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: partition_filters: 0 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: use_delta_encoding: 1 2024-09-20T18:49:25.405 INFO:journalctl@ceph.mon.a.smithi076.stdout: filter_policy: bloomfilter 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: whole_key_filtering: 1 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: verify_compression: 0 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: read_amp_bytes_per_bit: 0 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: format_version: 5 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_index_compression: 1 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_align: 0 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: max_auto_readahead_size: 262144 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: prepopulate_block_cache: 0 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: initial_auto_readahead_size: 8192 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression: NoCompression 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T18:49:25.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.num_levels: 7 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T18:49:25.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T18:49:25.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T18:49:25.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T18:49:25.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.inplace_update_support: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.bloom_locality: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.max_successive_merges: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T18:49:25.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.ttl: 2592000 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.enable_blob_files: false 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.min_blob_size: 0 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T18:49:25.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5bd130db-f79f-4509-9780-198d4e4f37ed 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858165206102, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858165206979, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80297, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 78458, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 72769, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726858165, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5bd130db-f79f-4509-9780-198d4e4f37ed", "db_session_id": "8AA97AAX4YC40RPCVMTG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858165207054, "job": 1, "event": "recovery_finished"} 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b47ef2ee00 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: DB pointer 0x55b47f022000 2024-09-20T18:49:25.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** DB Stats ** 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-09-20T18:49:25.414 INFO:journalctl@ceph.mon.a.smithi076.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: L0 2/0 80.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: Sum 2/0 80.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T18:49:25.415 INFO:journalctl@ceph.mon.a.smithi076.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative compaction: 0.00 GB write, 24.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval compaction: 0.00 GB write, 24.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache BinnedLRUCache@0x55b47ef03350#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: starting mon.a rank 0 at public addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] at bind addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???) e1 preinit fsid eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).mds e1 new map 2024-09-20T18:49:25.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).mds e1 print_map 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout: e1 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout: legacy client fscid: -1 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout: No filesystems configured 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 bash[23629]: 856419dcdf1284f6960ad3b9b84af688230c38c666727a5c3187907570638b00 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 systemd[1]: Started Ceph mon.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:49:25.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T18:49:25.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: monmap e1: 1 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:49:25.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: fsmap 2024-09-20T18:49:25.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T18:49:25.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:25 smithi076 ceph-mon[23648]: mgrmap e1: no daemons active 2024-09-20T18:49:26.501 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-20T18:49:26.532 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T18:49:26.532 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mgr... 2024-09-20T18:49:26.532 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-20T18:49:26.532 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-20T18:49:26.790 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y 2024-09-20T18:49:26.791 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y.service: Unit ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y.service not loaded. 2024-09-20T18:49:26.938 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eefb3510-7780-11ef-baf5-efdc52797490.target.wants/ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-eefb3510-7780-11ef-baf5-efdc52797490@.service. 2024-09-20T18:49:27.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:26 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1943502643' entity='client.admin' 2024-09-20T18:49:27.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:26 smithi076 systemd[1]: Starting Ceph mgr.y for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:49:27.486 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 podman[23862]: 2024-09-20 18:49:27.052581696 +0000 UTC m=+0.015524892 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T18:49:27.487 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 podman[23862]: 2024-09-20 18:49:27.168018378 +0000 UTC m=+0.130961610 container create db9f4b4ab948694b60f2f7dc5f1e8a7d59b54d6e8dda945e5a4f579cc9213668 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0) 2024-09-20T18:49:27.487 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 podman[23862]: 2024-09-20 18:49:27.370166319 +0000 UTC m=+0.333109524 container init db9f4b4ab948694b60f2f7dc5f1e8a7d59b54d6e8dda945e5a4f579cc9213668 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T18:49:27.487 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 podman[23862]: 2024-09-20 18:49:27.372891098 +0000 UTC m=+0.335834279 container start db9f4b4ab948694b60f2f7dc5f1e8a7d59b54d6e8dda945e5a4f579cc9213668 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-20T18:49:27.487 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 bash[23862]: db9f4b4ab948694b60f2f7dc5f1e8a7d59b54d6e8dda945e5a4f579cc9213668 2024-09-20T18:49:27.500 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T18:49:27.500 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-09-20T18:49:27.507 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T18:49:27.507 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-09-20T18:49:27.507 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-09-20T18:49:27.507 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T18:49:27.516 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T18:49:27.516 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-09-20T18:49:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T18:49:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-09-20T18:49:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-09-20T18:49:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-20T18:49:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr to start... 2024-09-20T18:49:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr... 2024-09-20T18:49:27.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 systemd[1]: Started Ceph mgr.y for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:49:27.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:27.491+0000 7ff38e043040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T18:49:27.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:27.552+0000 7ff38e043040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T18:49:27.790 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:27.704+0000 7ff38e043040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T18:49:27.790 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:27.772+0000 7ff38e043040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T18:49:28.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:27 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:27.922+0000 7ff38e043040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:49:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:28 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/743583993' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T18:49:28.343 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:28 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:28.071+0000 7ff38e043040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T18:49:28.643 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:28 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:28.383+0000 7ff38e043040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T18:49:28.643 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:28 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:28.443+0000 7ff38e043040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "eefb3510-7780-11ef-baf5-efdc52797490", 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T18:49:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T18:49:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T18:49:28.661 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T18:49:28.662 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T18:49:18.761089+0000", 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:49:28.663 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (1/15)... 2024-09-20T18:49:29.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:28 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:28.626+0000 7ff38e043040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T18:49:29.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:29 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:29.487+0000 7ff38e043040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T18:49:29.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:29 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:29.551+0000 7ff38e043040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T18:49:30.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.258+0000 7ff38e043040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T18:49:30.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.370+0000 7ff38e043040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T18:49:30.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.507+0000 7ff38e043040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:49:30.866 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.568+0000 7ff38e043040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T18:49:30.867 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.718+0000 7ff38e043040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T18:49:30.867 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.788+0000 7ff38e043040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T18:49:30.867 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:30.849+0000 7ff38e043040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T18:49:31.141 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:31.011+0000 7ff38e043040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T18:49:31.141 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:31.080+0000 7ff38e043040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T18:49:31.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1116259827' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T18:49:31.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:31.153+0000 7ff38e043040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T18:49:31.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:31.240+0000 7ff38e043040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T18:49:31.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:31.302+0000 7ff38e043040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T18:49:31.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:31 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:31.364+0000 7ff38e043040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T18:49:32.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:32.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:49:32.041 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "eefb3510-7780-11ef-baf5-efdc52797490", 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-20T18:49:32.042 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T18:49:32.043 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.044 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T18:49:18.761089+0000", 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:32.045 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T18:49:32.046 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:49:32.046 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (2/15)... 2024-09-20T18:49:32.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:32.178+0000 7ff38e043040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T18:49:32.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: Activating manager daemon y 2024-09-20T18:49:32.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: mgrmap e2: y(active, starting, since 0.00303435s) 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: Manager daemon y is now available 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' 2024-09-20T18:49:32.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:32 smithi076 ceph-mon[23648]: from='mgr.14100 172.21.15.76:0/1193431117' entity='mgr.y' 2024-09-20T18:49:34.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:34 smithi076 ceph-mon[23648]: mgrmap e3: y(active, since 1.00626s) 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "eefb3510-7780-11ef-baf5-efdc52797490", 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T18:49:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T18:49:35.205 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:35.205 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T18:49:35.205 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T18:49:35.205 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T18:49:35.205 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T18:49:35.208 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T18:49:35.208 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T18:49:35.208 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.208 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T18:49:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T18:49:18.761089+0000", 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:49:35.211 INFO:teuthology.orchestra.run.smithi076.stdout:mgr is available 2024-09-20T18:49:35.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:35 smithi076 ceph-mon[23648]: mgrmap e4: y(active, since 2s) 2024-09-20T18:49:35.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:35 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/130191678' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:36.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T18:49:36.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T18:49:36.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-09-20T18:49:36.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T18:49:36.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T18:49:36.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T18:49:36.326 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling cephadm module... 2024-09-20T18:49:36.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:36 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1415012823' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T18:49:36.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:36 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1415012823' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T18:49:37.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1543718962' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T18:49:37.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ignoring --setuser ceph since I am not root 2024-09-20T18:49:37.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ignoring --setgroup ceph since I am not root 2024-09-20T18:49:37.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:37.327+0000 7f5f19d52040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T18:49:37.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:37.382+0000 7f5f19d52040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T18:49:37.836 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:37.531+0000 7f5f19d52040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T18:49:37.836 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:37.598+0000 7f5f19d52040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T18:49:37.837 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:37.743+0000 7f5f19d52040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:49:38.131 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:37.887+0000 7f5f19d52040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T18:49:38.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:38 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1543718962' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T18:49:38.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:38 smithi076 ceph-mon[23648]: mgrmap e5: y(active, since 5s) 2024-09-20T18:49:38.430 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:38.205+0000 7f5f19d52040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T18:49:38.430 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:38.265+0000 7f5f19d52040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T18:49:38.725 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:38.451+0000 7f5f19d52040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:49:39.004 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-09-20T18:49:39.005 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 5... 2024-09-20T18:49:39.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:39 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4291172896' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T18:49:39.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:39 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:39.342+0000 7f5f19d52040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T18:49:39.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:39 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:39.410+0000 7f5f19d52040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T18:49:40.386 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.117+0000 7f5f19d52040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T18:49:40.386 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.231+0000 7f5f19d52040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T18:49:40.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.369+0000 7f5f19d52040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:49:40.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.429+0000 7f5f19d52040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T18:49:40.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.572+0000 7f5f19d52040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T18:49:40.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.641+0000 7f5f19d52040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T18:49:40.935 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.700+0000 7f5f19d52040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T18:49:40.935 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.853+0000 7f5f19d52040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T18:49:40.935 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.918+0000 7f5f19d52040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T18:49:41.197 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:40.985+0000 7f5f19d52040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T18:49:41.197 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:41.066+0000 7f5f19d52040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T18:49:41.197 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:41.123+0000 7f5f19d52040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T18:49:41.197 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:41.180+0000 7f5f19d52040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T18:49:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:41.969+0000 7f5f19d52040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T18:49:42.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: Active manager daemon y restarted 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: Activating manager daemon y 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: mgrmap e6: y(active, starting, since 0.003611s) 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: Manager daemon y is now available 2024-09-20T18:49:42.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:42.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:42.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:42.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:49:42.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:49:42.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T18:49:42.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:42 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T18:49:43.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-mon[23648]: Found migration_current of "None". Setting to last migration. 2024-09-20T18:49:43.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:43.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:43.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-mon[23648]: mgrmap e7: y(active, since 1.00551s) 2024-09-20T18:49:43.568 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:49:43.568 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-20T18:49:43.568 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T18:49:43.568 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:49:43.568 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 5 is available 2024-09-20T18:49:43.568 INFO:teuthology.orchestra.run.smithi076.stdout:Setting orchestrator backend to cephadm... 2024-09-20T18:49:44.204 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:44 smithi076 ceph-mon[23648]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:44 smithi076 ceph-mon[23648]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:44 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:43.809+0000 7f5e8727d640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:18:49:43] ENGINE Error in HTTPServer.serve 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: Traceback (most recent call last): 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._connections.run(self.expiration_interval) 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:49:44.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._run(expiration_interval) 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: s = self.context.wrap_socket( 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: return self.sslsocket_class._create( 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self.do_handshake() 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._sslobj.do_handshake() 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:49:44.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:45.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:49:43] ENGINE Bus STARTING 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:49:43] ENGINE Serving on https://172.21.15.76:7150 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:49:43] ENGINE Error in HTTPServer.serve 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: Traceback (most recent call last): 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._connections.run(self.expiration_interval) 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._run(expiration_interval) 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:49:45.184 INFO:journalctl@ceph.mon.a.smithi076.stdout: s = self.context.wrap_socket( 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: return self.sslsocket_class._create( 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: self.do_handshake() 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._sslobj.do_handshake() 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:49:43] ENGINE Serving on http://172.21.15.76:8765 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:49:43] ENGINE Bus STARTED 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: mgrmap e8: y(active, since 2s) 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:45.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:45 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:49:45.713 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-20T18:49:45.713 INFO:teuthology.orchestra.run.smithi076.stdout:Generating ssh key... 2024-09-20T18:49:46.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-mon[23648]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: Generating public/private rsa key pair. 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: Your identification has been saved in /tmp/tmpk06qb9l0/key 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: Your public key has been saved in /tmp/tmpk06qb9l0/key.pub 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: The key fingerprint is: 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: SHA256:qVH2CicpPTrI3zRB8Z8yVm4m/ipXVDETfT4LhVRaa3Y ceph-eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: The key's randomart image is: 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: +---[RSA 3072]----+ 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | . *=oo | 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | o .++.o| 2024-09-20T18:49:46.540 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | . + .. ..*E| 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | o + *.. .o.o| 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | . O S.B . o| 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | . . o @ B. . | 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | o o + o. | 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | . +.... | 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: | . .o... | 2024-09-20T18:49:46.541 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:46 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: +----[SHA256]-----+ 2024-09-20T18:49:47.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:47 smithi076 ceph-mon[23648]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:47.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:47 smithi076 ceph-mon[23648]: Generating ssh key... 2024-09-20T18:49:47.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:47 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:47.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:47 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:48.001 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvEMc+kJELT+3FG9pyQ7ZjPP3jPz9+Bl4YWS+0/rJEIqmkkVATgfJ58e/liPOZ2t/yk84r+V9AhWqVwlZdk0+YFmpbnRIpV1wyzkukBpBSkzQOfZnykuU3kUeN5CD/FrS+Sdj99YMJKuddisdaQ8KG9sR6cs20yDRQ84NFyDCIsoy9/zSzr45bcMd84RpC3t4DPfOY9f+qTMSLHOQxEo55w2tANWRdeP1p0XOey4NNgu8dBF8BtFyYum+G6p/8eKanTrdlcmF9eqBcd6oI4WJUZMtCJPJC9lJoGB0q31IaxCqULBtYvUbjrVwErC9pLbsbTamc5u7n54BqABy5FdXxfk8dC0jWYcVIReaXWCJRPyaw3UD+cSdWWm60dA0A/ejGX12quLCN6X3Oui98F+93AbaKulyUvpyz9n256kyJtcgkzpAdYXVN4rSK8lVdDN3du1Xsrz0O0HxS6upoESY7r0GfYykXHBpXJ1QRRHpzz/Cr0oHMLt7F82RzUyN10c= ceph-eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:49:48.001 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T18:49:48.001 INFO:teuthology.orchestra.run.smithi076.stdout:Adding key to root@localhost authorized_keys... 2024-09-20T18:49:48.012 INFO:teuthology.orchestra.run.smithi076.stdout:Adding host smithi076... 2024-09-20T18:49:48.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:48 smithi076 ceph-mon[23648]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:49.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:49 smithi076 ceph-mon[23648]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:50.359 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:50 smithi076 ceph-mon[23648]: Deploying cephadm binary to smithi076 2024-09-20T18:49:51.725 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Added host 'smithi076' with addr '172.21.15.76' 2024-09-20T18:49:51.725 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mon service... 2024-09-20T18:49:52.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:52 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:52.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:52 smithi076 ceph-mon[23648]: Added host smithi076 2024-09-20T18:49:52.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:52 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:49:52.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-20T18:49:52.976 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mgr service... 2024-09-20T18:49:53.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:53 smithi076 ceph-mon[23648]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:53.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:53 smithi076 ceph-mon[23648]: Saving service mon spec with placement count:5 2024-09-20T18:49:53.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:53 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:54.870 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:54 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:54.870 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:54 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:54.870 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:54 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:55.397 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-20T18:49:55.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:55 smithi076 ceph-mon[23648]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:49:55.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:55 smithi076 ceph-mon[23648]: Saving service mgr spec with placement count:2 2024-09-20T18:49:57.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:57.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:57.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:49:57.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' 2024-09-20T18:49:57.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi076", "caps": []}]: dispatch 2024-09-20T18:49:57.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='mgr.14118 172.21.15.76:0/1200659567' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi076", "caps": []}]': finished 2024-09-20T18:49:57.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:56 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3757857496' entity='client.admin' 2024-09-20T18:49:57.694 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling the dashboard module... 2024-09-20T18:49:58.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:58 smithi076 ceph-mon[23648]: Deploying daemon agent.smithi076 on smithi076 2024-09-20T18:49:58.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:58 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1882264427' entity='client.admin' 2024-09-20T18:49:59.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1558712210' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T18:49:59.380 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ignoring --setuser ceph since I am not root 2024-09-20T18:49:59.380 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ignoring --setgroup ceph since I am not root 2024-09-20T18:49:59.381 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:59.221+0000 7f436f466040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T18:49:59.381 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:59.278+0000 7f436f466040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T18:49:59.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:59.428+0000 7f436f466040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T18:49:59.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:59.495+0000 7f436f466040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T18:49:59.658 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:59.641+0000 7f436f466040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:49:59.960 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:49:59 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:49:59.789+0000 7f436f466040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T18:50:00.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:00 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1558712210' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T18:50:00.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:00 smithi076 ceph-mon[23648]: mgrmap e9: y(active, since 17s) 2024-09-20T18:50:00.290 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:00.110+0000 7f436f466040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T18:50:00.290 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:00.174+0000 7f436f466040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T18:50:00.686 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:00.362+0000 7f436f466040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-09-20T18:50:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 9... 2024-09-20T18:50:01.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:01 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4258779947' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T18:50:01.377 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:01 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:01.248+0000 7f436f466040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T18:50:01.377 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:01 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:01.314+0000 7f436f466040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T18:50:02.306 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.033+0000 7f436f466040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T18:50:02.306 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.147+0000 7f436f466040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T18:50:02.590 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.288+0000 7f436f466040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:50:02.591 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.350+0000 7f436f466040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T18:50:02.591 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.502+0000 7f436f466040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T18:50:02.591 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.573+0000 7f436f466040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T18:50:02.885 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.636+0000 7f436f466040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T18:50:02.885 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.799+0000 7f436f466040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T18:50:02.885 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.867+0000 7f436f466040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T18:50:03.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:02.936+0000 7f436f466040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T18:50:03.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:03.022+0000 7f436f466040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T18:50:03.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:03.083+0000 7f436f466040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T18:50:03.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:03.144+0000 7f436f466040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T18:50:04.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:03.934+0000 7f436f466040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: Active manager daemon y restarted 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: Activating manager daemon y 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: mgrmap e10: y(active, starting, since 0.00310277s) 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: Manager daemon y is now available 2024-09-20T18:50:04.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:04.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:04.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T18:50:04.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T18:50:05.507 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:05.458+0000 7f42d58f0640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:18:50:05] ENGINE Error in HTTPServer.serve 2024-09-20T18:50:05.507 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: Traceback (most recent call last): 2024-09-20T18:50:05.507 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._connections.run(self.expiration_interval) 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._run(expiration_interval) 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:50:05.508 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: s = self.context.wrap_socket( 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: return self.sslsocket_class._create( 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self.do_handshake() 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._sslobj.do_handshake() 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:50:05.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:50:05.511 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-09-20T18:50:05.511 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-20T18:50:05.511 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T18:50:05.511 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-09-20T18:50:05.511 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 9 is available 2024-09-20T18:50:05.511 INFO:teuthology.orchestra.run.smithi076.stdout:Generating a dashboard self-signed certificate... 2024-09-20T18:50:06.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-mon[23648]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T18:50:06.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-mon[23648]: mgrmap e11: y(active, since 1.00556s) 2024-09-20T18:50:06.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:05 smithi076 ceph-mon[23648]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T18:50:06.673 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-20T18:50:06.673 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial admin user... 2024-09-20T18:50:07.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: [20/Sep/2024:18:50:05] ENGINE Bus STARTING 2024-09-20T18:50:07.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: [20/Sep/2024:18:50:05] ENGINE Serving on http://172.21.15.76:8765 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: [20/Sep/2024:18:50:05] ENGINE Serving on https://172.21.15.76:7150 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: [20/Sep/2024:18:50:05] ENGINE Bus STARTED 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: [20/Sep/2024:18:50:05] ENGINE Error in HTTPServer.serve 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: Traceback (most recent call last): 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._connections.run(self.expiration_interval) 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._run(expiration_interval) 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:50:07.290 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: s = self.context.wrap_socket( 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: return self.sslsocket_class._create( 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: self.do_handshake() 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._sslobj.do_handshake() 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: mgrmap e12: y(active, since 2s) 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi076", "caps": []}]: dispatch 2024-09-20T18:50:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:07.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:07.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:08.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:07 smithi076 ceph-mon[23648]: Deploying daemon agent.smithi076 on smithi076 2024-09-20T18:50:08.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:08.062 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ORlujg.XKoqNeDdkxyzKe.Gp4gsycQDMMTQCAOaGpsYujDGTFQZ62", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726858207, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T18:50:08.062 INFO:teuthology.orchestra.run.smithi076.stdout:Fetching dashboard port number... 2024-09-20T18:50:09.190 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:08 smithi076 ceph-mon[23648]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:09.190 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:08 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2418536056' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T18:50:09.193 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 8443 2024-09-20T18:50:09.202 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T18:50:09.202 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-09-20T18:50:09.209 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T18:50:09.209 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-09-20T18:50:09.209 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-09-20T18:50:09.209 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T18:50:09.211 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph Dashboard is now available at: 2024-09-20T18:50:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: URL: https://smithi076.front.sepia.ceph.com:8443/ 2024-09-20T18:50:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: User: admin 2024-09-20T18:50:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: Password: djkqtogf7p 2024-09-20T18:50:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:09.212 INFO:teuthology.orchestra.run.smithi076.stdout:Saving cluster configuration to /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config directory 2024-09-20T18:50:09.212 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling autotune for osd_memory_target 2024-09-20T18:50:11.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:10 smithi076 ceph-mon[23648]: mgrmap e13: y(active, since 6s) 2024-09-20T18:50:11.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:10 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1685470262' entity='client.admin' 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout:Or, if you are only running a single cluster on this host: 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.421 INFO:teuthology.orchestra.run.smithi076.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: ceph telemetry on 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout:For more information see: 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:11.422 INFO:teuthology.orchestra.run.smithi076.stdout:Bootstrap complete. 2024-09-20T18:50:11.455 INFO:tasks.cephadm:Fetching config... 2024-09-20T18:50:11.455 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:50:11.455 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T18:50:11.471 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T18:50:11.471 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:50:11.471 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T18:50:11.527 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T18:50:11.527 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:50:11.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-20T18:50:11.657 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T18:50:11.657 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:50:11.657 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T18:50:11.716 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T18:50:11.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvEMc+kJELT+3FG9pyQ7ZjPP3jPz9+Bl4YWS+0/rJEIqmkkVATgfJ58e/liPOZ2t/yk84r+V9AhWqVwlZdk0+YFmpbnRIpV1wyzkukBpBSkzQOfZnykuU3kUeN5CD/FrS+Sdj99YMJKuddisdaQ8KG9sR6cs20yDRQ84NFyDCIsoy9/zSzr45bcMd84RpC3t4DPfOY9f+qTMSLHOQxEo55w2tANWRdeP1p0XOey4NNgu8dBF8BtFyYum+G6p/8eKanTrdlcmF9eqBcd6oI4WJUZMtCJPJC9lJoGB0q31IaxCqULBtYvUbjrVwErC9pLbsbTamc5u7n54BqABy5FdXxfk8dC0jWYcVIReaXWCJRPyaw3UD+cSdWWm60dA0A/ejGX12quLCN6X3Oui98F+93AbaKulyUvpyz9n256kyJtcgkzpAdYXVN4rSK8lVdDN3du1Xsrz0O0HxS6upoESY7r0GfYykXHBpXJ1QRRHpzz/Cr0oHMLt7F82RzUyN10c= ceph-eefb3510-7780-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T18:50:11.853 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvEMc+kJELT+3FG9pyQ7ZjPP3jPz9+Bl4YWS+0/rJEIqmkkVATgfJ58e/liPOZ2t/yk84r+V9AhWqVwlZdk0+YFmpbnRIpV1wyzkukBpBSkzQOfZnykuU3kUeN5CD/FrS+Sdj99YMJKuddisdaQ8KG9sR6cs20yDRQ84NFyDCIsoy9/zSzr45bcMd84RpC3t4DPfOY9f+qTMSLHOQxEo55w2tANWRdeP1p0XOey4NNgu8dBF8BtFyYum+G6p/8eKanTrdlcmF9eqBcd6oI4WJUZMtCJPJC9lJoGB0q31IaxCqULBtYvUbjrVwErC9pLbsbTamc5u7n54BqABy5FdXxfk8dC0jWYcVIReaXWCJRPyaw3UD+cSdWWm60dA0A/ejGX12quLCN6X3Oui98F+93AbaKulyUvpyz9n256kyJtcgkzpAdYXVN4rSK8lVdDN3du1Xsrz0O0HxS6upoESY7r0GfYykXHBpXJ1QRRHpzz/Cr0oHMLt7F82RzUyN10c= ceph-eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:11.866 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvEMc+kJELT+3FG9pyQ7ZjPP3jPz9+Bl4YWS+0/rJEIqmkkVATgfJ58e/liPOZ2t/yk84r+V9AhWqVwlZdk0+YFmpbnRIpV1wyzkukBpBSkzQOfZnykuU3kUeN5CD/FrS+Sdj99YMJKuddisdaQ8KG9sR6cs20yDRQ84NFyDCIsoy9/zSzr45bcMd84RpC3t4DPfOY9f+qTMSLHOQxEo55w2tANWRdeP1p0XOey4NNgu8dBF8BtFyYum+G6p/8eKanTrdlcmF9eqBcd6oI4WJUZMtCJPJC9lJoGB0q31IaxCqULBtYvUbjrVwErC9pLbsbTamc5u7n54BqABy5FdXxfk8dC0jWYcVIReaXWCJRPyaw3UD+cSdWWm60dA0A/ejGX12quLCN6X3Oui98F+93AbaKulyUvpyz9n256kyJtcgkzpAdYXVN4rSK8lVdDN3du1Xsrz0O0HxS6upoESY7r0GfYykXHBpXJ1QRRHpzz/Cr0oHMLt7F82RzUyN10c= ceph-eefb3510-7780-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T18:50:11.917 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvEMc+kJELT+3FG9pyQ7ZjPP3jPz9+Bl4YWS+0/rJEIqmkkVATgfJ58e/liPOZ2t/yk84r+V9AhWqVwlZdk0+YFmpbnRIpV1wyzkukBpBSkzQOfZnykuU3kUeN5CD/FrS+Sdj99YMJKuddisdaQ8KG9sR6cs20yDRQ84NFyDCIsoy9/zSzr45bcMd84RpC3t4DPfOY9f+qTMSLHOQxEo55w2tANWRdeP1p0XOey4NNgu8dBF8BtFyYum+G6p/8eKanTrdlcmF9eqBcd6oI4WJUZMtCJPJC9lJoGB0q31IaxCqULBtYvUbjrVwErC9pLbsbTamc5u7n54BqABy5FdXxfk8dC0jWYcVIReaXWCJRPyaw3UD+cSdWWm60dA0A/ejGX12quLCN6X3Oui98F+93AbaKulyUvpyz9n256kyJtcgkzpAdYXVN4rSK8lVdDN3du1Xsrz0O0HxS6upoESY7r0GfYykXHBpXJ1QRRHpzz/Cr0oHMLt7F82RzUyN10c= ceph-eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:11.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T18:50:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:50:13.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:13.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:13.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:13.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:14.213 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:13 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:14.213 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:13 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/439661405' entity='client.admin' 2024-09-20T18:50:14.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:13 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:14.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:13 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:14.879 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T18:50:14.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T18:50:15.159 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:14 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:16.476 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:50:18.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:18.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:19.248 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-09-20T18:50:19.248 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:50:19.248 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-09-20T18:50:19.267 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:50:19.268 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:50:19.326 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-09-20T18:50:19.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch host add smithi194 2024-09-20T18:50:19.524 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:50:19.928 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:19.928 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:50:19.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:19.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:19.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:19.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:19.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:50:19.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:19.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:19.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:20.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:20 smithi076 ceph-mon[23648]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:50:22.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:22 smithi076 ceph-mon[23648]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:22.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:22 smithi076 ceph-mon[23648]: Deploying cephadm binary to smithi194 2024-09-20T18:50:23.833 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-09-20T18:50:24.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T18:50:24.610 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: Added host smithi194 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:25.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:50:25.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:25.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:50:25.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:50:25.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:25.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:25.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-09-20T18:50:25.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]': finished 2024-09-20T18:50:26.348 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:50:26.348 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-09-20T18:50:26.903 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T18:50:26.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T18:50:27.093 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:50:27.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: Deploying daemon agent.smithi194 on smithi194 2024-09-20T18:50:27.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:27.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:27.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:28.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:28 smithi076 ceph-mon[23648]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:50:28.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:28.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:29.114 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-09-20T18:50:29.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:29 smithi076 ceph-mon[23648]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:29.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:29 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2812133754' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T18:50:29.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:29 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:29.705 INFO:tasks.cephadm:Adding mon.a on smithi076 2024-09-20T18:50:29.706 INFO:tasks.cephadm:Adding mon.c on smithi076 2024-09-20T18:50:29.706 INFO:tasks.cephadm:Adding mon.b on smithi194 2024-09-20T18:50:29.706 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b' 2024-09-20T18:50:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:30.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:30 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2812133754' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T18:50:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:30 smithi076 ceph-mon[23648]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:30 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:30 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:31.409 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-09-20T18:50:31.441 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:31 smithi076 ceph-mon[23648]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:32.016 DEBUG:teuthology.orchestra.run.smithi076:mon.c> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.c.service 2024-09-20T18:50:32.020 DEBUG:teuthology.orchestra.run.smithi194:mon.b> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.b.service 2024-09-20T18:50:32.023 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T18:50:32.023 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T18:50:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b;count:3 2024-09-20T18:50:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: Deploying daemon mon.b on smithi194 2024-09-20T18:50:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:32 smithi076 ceph-mon[23648]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:32.791 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:34.984 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:50:34.984 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","modified":"2024-09-20T18:49:08.310978Z","created":"2024-09-20T18:49:08.310978Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T18:50:34.985 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-09-20T18:50:35.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:34 smithi076 ceph-mon[23648]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:35.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:34 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:35.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:34 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:35.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:34 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:35.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:34 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/504305582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:36.570 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T18:50:36.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T18:50:36.863 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:50:37.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:36 smithi076 ceph-mon[23648]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:38.477 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:50:38.478 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","modified":"2024-09-20T18:49:08.310978Z","created":"2024-09-20T18:49:08.310978Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T18:50:38.478 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-09-20T18:50:38.478 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 podman[25739]: 2024-09-20 18:50:38.286591327 +0000 UTC m=+0.015739571 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T18:50:39.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 podman[25739]: 2024-09-20 18:50:38.738423524 +0000 UTC m=+0.467571776 container init 0bafb7d000b40b3071e6343e92ee65bd80a35c6d1fdb7327fdfe9d9b47eea2dd (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 podman[25739]: 2024-09-20 18:50:38.742220582 +0000 UTC m=+0.471368835 container start 0bafb7d000b40b3071e6343e92ee65bd80a35c6d1fdb7327fdfe9d9b47eea2dd (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: pidfile_write: ignore empty --pid-file 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: load: jerasure load: lrc 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: RocksDB version: 7.9.2 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Git sha 0 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T18:50:39.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: DB SUMMARY 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: DB Session ID: NNWF1U7F8CAF29OXKZUL 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: CURRENT file: CURRENT 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.error_if_exists: 0 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.create_if_missing: 0 2024-09-20T18:50:39.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.paranoid_checks: 1 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.env: 0x55bc23650bc0 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.info_log: 0x55bc23edebe0 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.statistics: (nil) 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.use_fsync: 0 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_log_file_size: 0 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T18:50:39.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.allow_fallocate: 1 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.use_direct_reads: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.db_log_dir: 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.wal_dir: 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T18:50:39.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.write_buffer_manager: 0x55bc23eec320 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T18:50:39.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.unordered_write: 0 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.row_cache: None 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.wal_filter: None 2024-09-20T18:50:39.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T18:50:39.102 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.two_write_queues: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.wal_compression: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.atomic_flush: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.log_readahead_size: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T18:50:39.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_background_jobs: 2 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_background_compactions: -1 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_subcompactions: 1 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T18:50:39.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_open_files: -1 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_background_flushes: -1 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Compression algorithms supported: 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kZSTD supported: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kXpressCompression supported: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kBZip2Compression supported: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kLZ4Compression supported: 1 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kZlibCompression supported: 1 2024-09-20T18:50:39.105 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: kSnappyCompression supported: 1 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.merge_operator: 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_filter: None 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc23ede7e0) 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T18:50:39.106 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_type: 0 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_index_type: 0 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_shortening: 1 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: checksum: 4 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: no_block_cache: 0 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache: 0x55bc23ed3350 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_options: 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: capacity : 536870912 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_shard_bits : 4 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: strict_capacity_limit : 0 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_compressed: (nil) 2024-09-20T18:50:39.107 INFO:journalctl@ceph.mon.b.smithi194.stdout: persistent_cache: (nil) 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size: 4096 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size_deviation: 10 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_restart_interval: 16 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_block_restart_interval: 1 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: metadata_block_size: 4096 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: partition_filters: 0 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: use_delta_encoding: 1 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: filter_policy: bloomfilter 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: whole_key_filtering: 1 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: verify_compression: 0 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: format_version: 5 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_index_compression: 1 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_align: 0 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: max_auto_readahead_size: 262144 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: prepopulate_block_cache: 0 2024-09-20T18:50:39.108 INFO:journalctl@ceph.mon.b.smithi194.stdout: initial_auto_readahead_size: 8192 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression: NoCompression 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.num_levels: 7 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T18:50:39.109 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T18:50:39.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T18:50:39.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T18:50:39.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.inplace_update_support: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.bloom_locality: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.max_successive_merges: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T18:50:39.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.ttl: 2592000 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.enable_blob_files: false 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.min_blob_size: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T18:50:39.114 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e3b3d0f8-8114-468d-9e20-40ae48450238 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858238768967, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858238769369, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726858238, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e3b3d0f8-8114-468d-9e20-40ae48450238", "db_session_id": "NNWF1U7F8CAF29OXKZUL", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858238769462, "job": 1, "event": "recovery_finished"} 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bc23efee00 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: DB pointer 0x55bc23ff2000 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T18:50:39.115 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** DB Stats ** 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:39.116 INFO:journalctl@ceph.mon.b.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T18:50:39.117 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: Block cache BinnedLRUCache@0x55bc23ed3350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: using public_addr v2:172.21.15.194:0/0 -> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: starting mon.b rank -1 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(???) e0 preinit fsid eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).mds e1 new map 2024-09-20T18:50:39.118 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout: e1 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout: legacy client fscid: -1 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout: No filesystems configured 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T18:50:39.119 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2812133754' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2812133754' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:39.120 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b;count:3 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: Deploying daemon mon.b on smithi194 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.121 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:39.122 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/504305582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:39.122 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:39.122 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 ceph-mon[25770]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-20T18:50:39.122 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 bash[25739]: 0bafb7d000b40b3071e6343e92ee65bd80a35c6d1fdb7327fdfe9d9b47eea2dd 2024-09-20T18:50:39.122 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:38 smithi194 systemd[1]: Started Ceph mon.b for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:50:40.254 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T18:50:40.254 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T18:50:40.464 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: mon.a calling monitor election 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: mon.b calling monitor election 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: monmap e2: 2 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: fsmap 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: mgrmap e13: y(active, since 39s) 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: overall HEALTH_OK 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:44.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:44.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:43 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: mon.a calling monitor election 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: mon.b calling monitor election 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: monmap e2: 2 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:44.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: fsmap 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: mgrmap e13: y(active, since 39s) 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: overall HEALTH_OK 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:44.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:43 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:45.025 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:50:45.025 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","modified":"2024-09-20T18:50:38.793692Z","created":"2024-09-20T18:49:08.310978Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T18:50:45.025 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-09-20T18:50:45.209 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:44 smithi076 ceph-mon[23648]: Deploying daemon mon.c on smithi076 2024-09-20T18:50:45.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:44 smithi076 ceph-mon[23648]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:45.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:44 smithi194 ceph-mon[25770]: Deploying daemon mon.c on smithi076 2024-09-20T18:50:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:44 smithi194 ceph-mon[25770]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:46.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:45 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/3596681865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:46.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:45 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/3596681865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:46.634 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T18:50:46.635 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T18:50:46.851 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:50:46.972 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:46 smithi194 ceph-mon[25770]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:46 smithi076 ceph-mon[23648]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:46 smithi076 podman[30004]: 2024-09-20 18:50:46.897371666 +0000 UTC m=+0.020287139 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 podman[30004]: 2024-09-20 18:50:47.026116819 +0000 UTC m=+0.149032341 container create 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 podman[30004]: 2024-09-20 18:50:47.228370006 +0000 UTC m=+0.351285485 container init 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 podman[30004]: 2024-09-20 18:50:47.232840782 +0000 UTC m=+0.355756268 container start 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pidfile_write: ignore empty --pid-file 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: load: jerasure load: lrc 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: RocksDB version: 7.9.2 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Git sha 0 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: DB SUMMARY 2024-09-20T18:50:47.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: DB Session ID: E4XZC1NDUX63AO8LZ1CO 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: CURRENT file: CURRENT 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.error_if_exists: 0 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.create_if_missing: 0 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.paranoid_checks: 1 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.env: 0x5620cf63abc0 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T18:50:47.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.info_log: 0x5620d12aebe0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.statistics: (nil) 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.use_fsync: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_log_file_size: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.allow_fallocate: 1 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.use_direct_reads: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T18:50:47.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.db_log_dir: 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.wal_dir: 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.write_buffer_manager: 0x5620d12bc320 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T18:50:47.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T18:50:47.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T18:50:47.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T18:50:47.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T18:50:47.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T18:50:47.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T18:50:47.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.unordered_write: 0 2024-09-20T18:50:47.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T18:50:47.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T18:50:47.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T18:50:47.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T18:50:47.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.row_cache: None 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.wal_filter: None 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.two_write_queues: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.wal_compression: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.atomic_flush: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.log_readahead_size: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T18:50:47.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_background_jobs: 2 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_background_compactions: -1 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_subcompactions: 1 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T18:50:47.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_open_files: -1 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_background_flushes: -1 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Compression algorithms supported: 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kZSTD supported: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kXpressCompression supported: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kBZip2Compression supported: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kLZ4Compression supported: 1 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kZlibCompression supported: 1 2024-09-20T18:50:47.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: kSnappyCompression supported: 1 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.merge_operator: 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_filter: None 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5620d12ae7e0) 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout: cache_index_and_filter_blocks: 1 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T18:50:47.304 INFO:journalctl@ceph.mon.c.smithi076.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: pin_top_level_index_and_filter: 1 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: index_type: 0 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: data_block_index_type: 0 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: index_shortening: 1 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: checksum: 4 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: no_block_cache: 0 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_cache: 0x5620d12a3350 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_cache_name: BinnedLRUCache 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_cache_options: 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: capacity : 536870912 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: num_shard_bits : 4 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: strict_capacity_limit : 0 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: high_pri_pool_ratio: 0.000 2024-09-20T18:50:47.305 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_cache_compressed: (nil) 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: persistent_cache: (nil) 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_size: 4096 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_size_deviation: 10 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_restart_interval: 16 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: index_block_restart_interval: 1 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: metadata_block_size: 4096 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: partition_filters: 0 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: use_delta_encoding: 1 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: filter_policy: bloomfilter 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: whole_key_filtering: 1 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: verify_compression: 0 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: read_amp_bytes_per_bit: 0 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: format_version: 5 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: enable_index_compression: 1 2024-09-20T18:50:47.306 INFO:journalctl@ceph.mon.c.smithi076.stdout: block_align: 0 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout: max_auto_readahead_size: 262144 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout: prepopulate_block_cache: 0 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout: initial_auto_readahead_size: 8192 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression: NoCompression 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.num_levels: 7 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T18:50:47.307 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T18:50:47.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T18:50:47.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T18:50:47.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.inplace_update_support: 0 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.bloom_locality: 0 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.max_successive_merges: 0 2024-09-20T18:50:47.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.ttl: 2592000 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.enable_blob_files: false 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.min_blob_size: 0 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T18:50:47.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c970fd64-40bf-4325-95e6-6bc4a62aa160 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858247259657, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858247260086, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726858247, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c970fd64-40bf-4325-95e6-6bc4a62aa160", "db_session_id": "E4XZC1NDUX63AO8LZ1CO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726858247260181, "job": 1, "event": "recovery_finished"} 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5620d12cee00 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: DB pointer 0x5620d13c2000 2024-09-20T18:50:47.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T18:50:47.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T18:50:47.314 INFO:journalctl@ceph.mon.c.smithi076.stdout: ** DB Stats ** 2024-09-20T18:50:47.314 INFO:journalctl@ceph.mon.c.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T18:50:47.314 INFO:journalctl@ceph.mon.c.smithi076.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T18:50:47.314 INFO:journalctl@ceph.mon.c.smithi076.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: ** Compaction Stats [default] ** 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: ** Compaction Stats [default] ** 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T18:50:47.315 INFO:journalctl@ceph.mon.c.smithi076.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T18:50:47.316 INFO:journalctl@ceph.mon.c.smithi076.stdout: Block cache BinnedLRUCache@0x5620d12a3350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: using public_addrv [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: starting mon.c rank -1 at public addrs [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] at bind addrs [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(???) e0 preinit fsid eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).mds e1 new map 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: e1 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: legacy client fscid: -1 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: 2024-09-20T18:50:47.317 INFO:journalctl@ceph.mon.c.smithi076.stdout: No filesystems configured 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2812133754' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2812133754' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T18:50:47.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c;smithi194:172.21.15.194=b;count:3 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: Deploying daemon mon.b on smithi194 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/504305582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.a calling monitor election 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.b calling monitor election 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: monmap e2: 2 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: fsmap 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mgrmap e13: y(active, since 39s) 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: overall HEALTH_OK 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: Deploying daemon mon.c on smithi076 2024-09-20T18:50:47.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.322 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:47.322 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/3596681865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:47.322 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:47.322 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 ceph-mon[30031]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-20T18:50:47.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 bash[30004]: 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 2024-09-20T18:50:47.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:47 smithi076 systemd[1]: Started Ceph mon.c for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: mon.a calling monitor election 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: mon.b calling monitor election 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:52.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: fsmap 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: mgrmap e13: y(active, since 48s) 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T18:50:52.744 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: mon.c (rank 2) addr [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] is down (out of quorum) 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:50:52.745 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","modified":"2024-09-20T18:50:47.283258Z","created":"2024-09-20T18:49:08.310978Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3301","nonce":0},{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T18:50:52.746 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: mon.a calling monitor election 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: mon.b calling monitor election 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: fsmap 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: mgrmap e13: y(active, since 48s) 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: mon.c (rank 2) addr [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] is down (out of quorum) 2024-09-20T18:50:52.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:52.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.331 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T18:50:53.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/2959455736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:50:53.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: Reconfiguring daemon mon.a on smithi076 2024-09-20T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:53.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/2959455736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:53.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: Reconfiguring daemon mon.a on smithi076 2024-09-20T18:50:53.593 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:53.955 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.a calling monitor election 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.b calling monitor election 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: fsmap 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mgrmap e13: y(active, since 48s) 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T18:50:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.c (rank 2) addr [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] is down (out of quorum) 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/2959455736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Reconfiguring daemon mon.a on smithi076 2024-09-20T18:50:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:55.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: mon.c calling monitor election 2024-09-20T18:50:55.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: mon.c calling monitor election 2024-09-20T18:50:55.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: mon.b calling monitor election 2024-09-20T18:50:55.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: mon.a calling monitor election 2024-09-20T18:50:55.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T18:50:55.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:55.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: fsmap 2024-09-20T18:50:55.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:55.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: mgrmap e13: y(active, since 50s) 2024-09-20T18:50:55.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T18:50:55.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: Cluster is now healthy 2024-09-20T18:50:55.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:54 smithi194 ceph-mon[25770]: overall HEALTH_OK 2024-09-20T18:50:55.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: mon.c calling monitor election 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: mon.c calling monitor election 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: mon.b calling monitor election 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: mon.a calling monitor election 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: fsmap 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: mgrmap e13: y(active, since 50s) 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: Cluster is now healthy 2024-09-20T18:50:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[23648]: overall HEALTH_OK 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.c calling monitor election 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.c calling monitor election 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.b calling monitor election 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.a calling monitor election 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: fsmap 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: mgrmap e13: y(active, since 50s) 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: Cluster is now healthy 2024-09-20T18:50:55.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:54 smithi076 ceph-mon[30031]: overall HEALTH_OK 2024-09-20T18:50:56.103 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.104 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:55 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:56.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:50:56.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:50:56.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:56.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:56.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:55 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:50:57.020 INFO:teuthology.orchestra.run.smithi076.stdout:# minimal ceph.conf for eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:57.021 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-09-20T18:50:57.021 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T18:50:57.021 INFO:teuthology.orchestra.run.smithi076.stdout: mon_host = [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] 2024-09-20T18:50:57.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:56 smithi076 ceph-mon[23648]: Reconfiguring mon.c (monmap changed)... 2024-09-20T18:50:57.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:56 smithi076 ceph-mon[23648]: Reconfiguring daemon mon.c on smithi076 2024-09-20T18:50:57.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:56 smithi076 ceph-mon[23648]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:57.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:56 smithi076 ceph-mon[30031]: Reconfiguring mon.c (monmap changed)... 2024-09-20T18:50:57.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:56 smithi076 ceph-mon[30031]: Reconfiguring daemon mon.c on smithi076 2024-09-20T18:50:57.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:56 smithi076 ceph-mon[30031]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:57.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:56 smithi194 ceph-mon[25770]: Reconfiguring mon.c (monmap changed)... 2024-09-20T18:50:57.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:56 smithi194 ceph-mon[25770]: Reconfiguring daemon mon.c on smithi076 2024-09-20T18:50:57.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:56 smithi194 ceph-mon[25770]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:57.648 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T18:50:57.648 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:50:57.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T18:50:57.676 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:50:57.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:50:57.743 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:50:57.743 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T18:50:57.778 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:50:57.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:50:57.849 INFO:tasks.cephadm:Adding mgr.y on smithi076 2024-09-20T18:50:57.849 INFO:tasks.cephadm:Adding mgr.x on smithi194 2024-09-20T18:50:57.849 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch apply mgr '2;smithi076=y;smithi194=x' 2024-09-20T18:50:58.083 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:50:58.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:57 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1927633143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:58.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:57 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1927633143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:58.256 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:57 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1927633143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:50:59.189 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:50:58 smithi194 ceph-mon[25770]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:59.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:50:58 smithi076 ceph-mon[23648]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:59.221 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:50:58 smithi076 ceph-mon[30031]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:50:59.557 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-09-20T18:51:00.115 DEBUG:teuthology.orchestra.run.smithi194:mgr.x> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.x.service 2024-09-20T18:51:00.117 DEBUG:tasks.cephadm:set 0 configs 2024-09-20T18:51:00.117 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T18:51:00.117 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:51:00.117 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T18:51:00.134 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T18:51:00.134 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-20 18:50:57.296940284 +0000 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-20 18:48:10.727580278 +0000 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-20 18:48:10.727580278 +0000 2024-09-20T18:51:00.191 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-20 18:48:10.727580278 +0000 2024-09-20T18:51:00.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T18:51:00.256 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-20T18:51:00.256 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-20T18:51:00.257 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000202653 s, 2.5 MB/s 2024-09-20T18:51:00.257 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T18:51:00.314 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-09-20T18:51:00.369 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T18:51:00.369 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:00.369 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-20T18:51:00.370 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:00.370 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:00.370 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-20 18:50:57.296940284 +0000 2024-09-20T18:51:00.370 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-20 18:48:11.060570257 +0000 2024-09-20T18:51:00.370 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-20 18:48:11.060570257 +0000 2024-09-20T18:51:00.370 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-20 18:48:11.060570257 +0000 2024-09-20T18:51:00.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T18:51:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-20T18:51:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-20T18:51:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000190908 s, 2.7 MB/s 2024-09-20T18:51:00.437 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T18:51:00.496 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-09-20T18:51:00.551 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T18:51:00.551 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:00.551 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-20T18:51:00.552 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:00.552 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:00.552 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-20 18:50:57.296940284 +0000 2024-09-20T18:51:00.552 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-20 18:48:11.384560507 +0000 2024-09-20T18:51:00.552 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-20 18:48:11.384560507 +0000 2024-09-20T18:51:00.552 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-20 18:48:11.384560507 +0000 2024-09-20T18:51:00.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T18:51:00.617 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-20T18:51:00.617 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-20T18:51:00.618 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000233754 s, 2.2 MB/s 2024-09-20T18:51:00.619 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T18:51:00.675 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-09-20T18:51:00.731 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T18:51:00.731 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:00.731 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-20T18:51:00.731 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:00.731 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:00.731 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-20 18:50:57.296940284 +0000 2024-09-20T18:51:00.732 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-20 18:48:11.700550997 +0000 2024-09-20T18:51:00.732 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-20 18:48:11.700550997 +0000 2024-09-20T18:51:00.732 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-20 18:48:11.700550997 +0000 2024-09-20T18:51:00.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T18:51:00.795 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-20T18:51:00.796 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-20T18:51:00.796 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000187741 s, 2.7 MB/s 2024-09-20T18:51:00.797 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi076=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: Saving service mgr spec with placement smithi076=y;smithi194=x;count:2 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:00 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi076=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: Saving service mgr spec with placement smithi076=y;smithi194=x;count:2 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi076=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: Saving service mgr spec with placement smithi076=y;smithi194=x;count:2 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:00.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T18:51:00.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T18:51:00.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:00 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:00.829 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:51:00.829 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T18:51:00.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T18:51:00.845 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-09-20T18:51:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-20 18:50:55.148662754 +0000 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-20 18:48:10.732976528 +0000 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-20 18:48:10.732976528 +0000 2024-09-20T18:51:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-20 18:48:10.732976528 +0000 2024-09-20T18:51:00.901 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T18:51:00.967 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-20T18:51:00.967 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-20T18:51:00.967 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000210967 s, 2.4 MB/s 2024-09-20T18:51:00.968 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T18:51:01.025 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-20 18:50:55.148662754 +0000 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-20 18:48:11.054969893 +0000 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-20 18:48:11.054969893 +0000 2024-09-20T18:51:01.081 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-20 18:48:11.054969893 +0000 2024-09-20T18:51:01.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T18:51:01.144 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-20T18:51:01.145 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-20T18:51:01.145 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000202196 s, 2.5 MB/s 2024-09-20T18:51:01.146 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T18:51:01.202 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-20 18:50:55.148662754 +0000 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-20 18:48:11.384963097 +0000 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-20 18:48:11.384963097 +0000 2024-09-20T18:51:01.257 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-20 18:48:11.384963097 +0000 2024-09-20T18:51:01.258 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T18:51:01.322 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-20T18:51:01.323 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-20T18:51:01.323 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000184141 s, 2.8 MB/s 2024-09-20T18:51:01.323 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T18:51:01.379 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-09-20T18:51:01.434 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T18:51:01.434 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T18:51:01.434 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-20T18:51:01.434 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T18:51:01.434 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T18:51:01.435 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-20 18:50:55.148662754 +0000 2024-09-20T18:51:01.435 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-20 18:48:11.716956262 +0000 2024-09-20T18:51:01.435 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-20 18:48:11.716956262 +0000 2024-09-20T18:51:01.435 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-20 18:48:11.716956262 +0000 2024-09-20T18:51:01.435 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T18:51:01.498 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-20T18:51:01.498 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-20T18:51:01.498 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000196957 s, 2.6 MB/s 2024-09-20T18:51:01.499 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T18:51:01.555 INFO:tasks.cephadm:Deploying osd.0 on smithi076 with /dev/vg_nvme/lv_4... 2024-09-20T18:51:01.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T18:51:01.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:01 smithi076 ceph-mon[30031]: Reconfiguring mon.b (monmap changed)... 2024-09-20T18:51:01.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:01 smithi076 ceph-mon[30031]: Reconfiguring daemon mon.b on smithi194 2024-09-20T18:51:01.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:01 smithi076 ceph-mon[23648]: Reconfiguring mon.b (monmap changed)... 2024-09-20T18:51:01.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:01 smithi076 ceph-mon[23648]: Reconfiguring daemon mon.b on smithi194 2024-09-20T18:51:01.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:01 smithi194 ceph-mon[25770]: Reconfiguring mon.b (monmap changed)... 2024-09-20T18:51:01.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:01 smithi194 ceph-mon[25770]: Reconfiguring daemon mon.b on smithi194 2024-09-20T18:51:01.910 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:51:03.023 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: Deploying daemon mgr.x on smithi194 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:03.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.025 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T18:51:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: Deploying daemon mgr.x on smithi194 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: Deploying daemon mgr.x on smithi194 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:03.595 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 systemd[1]: Starting Ceph mgr.x for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 podman[27936]: 2024-09-20 18:51:03.567242382 +0000 UTC m=+0.015636912 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 podman[27936]: 2024-09-20 18:51:03.675371894 +0000 UTC m=+0.123766421 container create 7b8f27eb749782df3f2b1852d9b98388c059af652648324dc3c78f583ee49c04 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 podman[27936]: 2024-09-20 18:51:03.836137828 +0000 UTC m=+0.284532366 container init 7b8f27eb749782df3f2b1852d9b98388c059af652648324dc3c78f583ee49c04 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 podman[27936]: 2024-09-20 18:51:03.839403057 +0000 UTC m=+0.287797585 container start 7b8f27eb749782df3f2b1852d9b98388c059af652648324dc3c78f583ee49c04 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 ceph-mgr[27954]: -- 172.21.15.194:0/1669522230 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561a5a378d00 con 0x561a5a34d000 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 bash[27936]: 7b8f27eb749782df3f2b1852d9b98388c059af652648324dc3c78f583ee49c04 2024-09-20T18:51:03.987 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 systemd[1]: Started Ceph mgr.x for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:51:03.988 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:03.969+0000 7f90b5ccc040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T18:51:04.215 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:51:04.234 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2024-09-20T18:51:04.264 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.029+0000 7f90b5ccc040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T18:51:04.264 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.182+0000 7f90b5ccc040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T18:51:04.264 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.247+0000 7f90b5ccc040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T18:51:04.542 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.388+0000 7f90b5ccc040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:51:04.542 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.525+0000 7f90b5ccc040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T18:51:05.098 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.827+0000 7f90b5ccc040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T18:51:05.098 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:04.886+0000 7f90b5ccc040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T18:51:05.098 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T18:51:05.099 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T18:51:05.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:05.100 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:04 smithi194 ceph-mon[25770]: Reconfiguring daemon mgr.y on smithi076 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T18:51:05.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[23648]: Reconfiguring daemon mgr.y on smithi076 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:05.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:05.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:04 smithi076 ceph-mon[30031]: Reconfiguring daemon mgr.y on smithi076 2024-09-20T18:51:05.380 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:05.086+0000 7f90b5ccc040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T18:51:05.524 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:51:06.335 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:05.978+0000 7f90b5ccc040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T18:51:06.335 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:06.044+0000 7f90b5ccc040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:51:06.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[30031]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:51:06.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:06 smithi076 ceph-mon[23648]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:06.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:06.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:06.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:06.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:06.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T18:51:06.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-mon[25770]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:07.131 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:06.842+0000 7f90b5ccc040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T18:51:07.132 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:06.965+0000 7f90b5ccc040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T18:51:07.132 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.114+0000 7f90b5ccc040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:51:07.409 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.178+0000 7f90b5ccc040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T18:51:07.409 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.324+0000 7f90b5ccc040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T18:51:07.409 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.392+0000 7f90b5ccc040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T18:51:07.693 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-mon[25770]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:07.694 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:07.694 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:07.694 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:07.694 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.451+0000 7f90b5ccc040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T18:51:07.694 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.610+0000 7f90b5ccc040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T18:51:07.694 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.676+0000 7f90b5ccc040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T18:51:07.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[23648]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:07.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:07.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:07.714 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:07.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[30031]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:07.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:07.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:07.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:07.968 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.743+0000 7f90b5ccc040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T18:51:07.968 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.830+0000 7f90b5ccc040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T18:51:07.968 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.891+0000 7f90b5ccc040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T18:51:07.968 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:07.950+0000 7f90b5ccc040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T18:51:09.095 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:51:08.743+0000 7f90b5ccc040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T18:51:09.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: Standby manager daemon x started 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T18:51:09.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:08 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: Standby manager daemon x started 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: Standby manager daemon x started 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T18:51:09.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:08 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/379762160' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3622319766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "802eb93b-c4da-44ae-ab87-912b6094a939"}]: dispatch 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3622319766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "802eb93b-c4da-44ae-ab87-912b6094a939"}]': finished 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T18:51:10.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4208387128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3622319766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "802eb93b-c4da-44ae-ab87-912b6094a939"}]: dispatch 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3622319766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "802eb93b-c4da-44ae-ab87-912b6094a939"}]': finished 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:10.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:09 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4208387128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:10.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:10.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-20T18:51:10.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T18:51:10.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3622319766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "802eb93b-c4da-44ae-ab87-912b6094a939"}]: dispatch 2024-09-20T18:51:10.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3622319766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "802eb93b-c4da-44ae-ab87-912b6094a939"}]': finished 2024-09-20T18:51:10.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T18:51:10.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:10.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:09 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4208387128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:11.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:10 smithi076 ceph-mon[23648]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:11.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:10 smithi076 ceph-mon[30031]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:11.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:10 smithi194 ceph-mon[25770]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:13.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:12 smithi076 ceph-mon[23648]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:13.211 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:12 smithi076 ceph-mon[30031]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:13.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:12 smithi194 ceph-mon[25770]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:15.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:14 smithi076 ceph-mon[23648]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:15.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:14 smithi076 ceph-mon[30031]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:15.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:14 smithi194 ceph-mon[25770]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:16.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:15 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T18:51:16.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:15 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:16.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:15 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T18:51:16.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:15 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:16.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:15 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T18:51:16.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:15 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:17.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:17 smithi076 ceph-mon[30031]: Deploying daemon osd.0 on smithi076 2024-09-20T18:51:17.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:17 smithi076 ceph-mon[30031]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:17.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:17 smithi076 ceph-mon[23648]: Deploying daemon osd.0 on smithi076 2024-09-20T18:51:17.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:17 smithi076 ceph-mon[23648]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:17.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:17 smithi194 ceph-mon[25770]: Deploying daemon osd.0 on smithi076 2024-09-20T18:51:17.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:17 smithi194 ceph-mon[25770]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:19.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:19 smithi076 ceph-mon[23648]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:19.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:19 smithi076 ceph-mon[30031]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:19.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:19 smithi194 ceph-mon[25770]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:21.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:21 smithi076 ceph-mon[23648]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:21.075 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:21 smithi076 ceph-mon[30031]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:21.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:21 smithi194 ceph-mon[25770]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:22.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:22.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:22.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:22.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:22.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:22.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:22.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:23.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:23.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:23.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:23.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:23.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:23.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:23.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.035 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 0 on host 'smithi076' 2024-09-20T18:51:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T18:51:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T18:51:25.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:25.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:25.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:25.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:25.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:25.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T18:51:25.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:25.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:25.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:25.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:25.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:25.646 DEBUG:teuthology.orchestra.run.smithi076:osd.0> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T18:51:25.649 INFO:tasks.cephadm:Deploying osd.1 on smithi076 with /dev/vg_nvme/lv_3... 2024-09-20T18:51:25.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T18:51:25.879 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[23648]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[23648]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[23648]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[30031]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[30031]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:26.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-mon[30031]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:51:26.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:26 smithi194 ceph-mon[25770]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T18:51:26.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:26 smithi194 ceph-mon[25770]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T18:51:26.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:26 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:26.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:26 smithi194 ceph-mon[25770]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:51:26.790 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0[34318]: 2024-09-20T18:51:26.513+0000 7f5126abc640 -1 osd.0 0 waiting for initial osdmap 2024-09-20T18:51:26.790 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:51:26 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0[34318]: 2024-09-20T18:51:26.515+0000 7f511e8dd640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:51:27.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' 2024-09-20T18:51:27.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:51:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T18:51:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:27.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970]' entity='osd.0' 2024-09-20T18:51:27.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:27.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:28.229 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:28 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:51:28.229 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:28 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:51:28.229 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:28 smithi194 ceph-mon[25770]: osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970] boot 2024-09-20T18:51:28.229 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:28 smithi194 ceph-mon[25770]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T18:51:28.229 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:28.229 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:28 smithi194 ceph-mon[25770]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:51:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:51:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[23648]: osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970] boot 2024-09-20T18:51:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[23648]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T18:51:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:28.342 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[23648]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:28.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:51:28.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:51:28.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[30031]: osd.0 [v2:172.21.15.76:6802/2779901970,v1:172.21.15.76:6803/2779901970] boot 2024-09-20T18:51:28.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[30031]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T18:51:28.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:51:28.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:28 smithi076 ceph-mon[30031]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T18:51:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:51:28.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2024-09-20T18:51:28.562 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:51:30.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:30 smithi076 ceph-mon[30031]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T18:51:30.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:30 smithi076 ceph-mon[30031]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:30 smithi076 ceph-mon[23648]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T18:51:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:30 smithi076 ceph-mon[23648]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:30.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:30 smithi194 ceph-mon[25770]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T18:51:30.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:30 smithi194 ceph-mon[25770]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:31.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[30031]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:31.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:31.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:31.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:31.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[23648]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:31.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:31.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:31.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:31 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:31.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:31 smithi194 ceph-mon[25770]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:31.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:31 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:31.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:31 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:31.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:31 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:32.496 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:32 smithi194 ceph-mon[25770]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:32.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:32 smithi076 ceph-mon[23648]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:32.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:32 smithi076 ceph-mon[30031]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:33.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4006807310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deafe02c-4749-4ca9-bc3b-5537831c731c"}]: dispatch 2024-09-20T18:51:33.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4006807310' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deafe02c-4749-4ca9-bc3b-5537831c731c"}]': finished 2024-09-20T18:51:33.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[23648]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T18:51:33.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:33.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2217454262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:33.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4006807310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deafe02c-4749-4ca9-bc3b-5537831c731c"}]: dispatch 2024-09-20T18:51:33.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4006807310' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deafe02c-4749-4ca9-bc3b-5537831c731c"}]': finished 2024-09-20T18:51:33.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[30031]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T18:51:33.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:33.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:33 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2217454262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:33 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4006807310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deafe02c-4749-4ca9-bc3b-5537831c731c"}]: dispatch 2024-09-20T18:51:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:33 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4006807310' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deafe02c-4749-4ca9-bc3b-5537831c731c"}]': finished 2024-09-20T18:51:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:33 smithi194 ceph-mon[25770]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T18:51:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:33 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2217454262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:34.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:34 smithi076 ceph-mon[23648]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:34.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:34 smithi076 ceph-mon[30031]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:34.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:34 smithi194 ceph-mon[25770]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi076 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:36.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi076 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:36.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:35 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi076 2024-09-20T18:51:36.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:36.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:36.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:36.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:36.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:35 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:37.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:36 smithi076 ceph-mon[23648]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:37.232 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:36 smithi076 ceph-mon[30031]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:37.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:36 smithi194 ceph-mon[25770]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:39.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:38 smithi076 ceph-mon[23648]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:39.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:38 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T18:51:39.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:38 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:39.077 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:38 smithi076 ceph-mon[30031]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:39.077 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:38 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T18:51:39.077 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:38 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:39.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:38 smithi194 ceph-mon[25770]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:39.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T18:51:39.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:40.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:39 smithi076 ceph-mon[23648]: Deploying daemon osd.1 on smithi076 2024-09-20T18:51:40.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:39 smithi076 ceph-mon[30031]: Deploying daemon osd.1 on smithi076 2024-09-20T18:51:40.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:39 smithi194 ceph-mon[25770]: Deploying daemon osd.1 on smithi076 2024-09-20T18:51:41.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:40 smithi076 ceph-mon[23648]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:41.242 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:40 smithi076 ceph-mon[30031]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:41.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:40 smithi194 ceph-mon[25770]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:43.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:42 smithi076 ceph-mon[23648]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:43.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:42 smithi076 ceph-mon[30031]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:42 smithi194 ceph-mon[25770]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:45.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:45.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:45.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:47.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:47 smithi194 ceph-mon[25770]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:47.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:47 smithi076 ceph-mon[23648]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:47.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:47 smithi076 ceph-mon[30031]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:48.005 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 1 on host 'smithi076' 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:48.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:48.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.515 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.515 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.515 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:48.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:48.649 DEBUG:teuthology.orchestra.run.smithi076:osd.1> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T18:51:48.652 INFO:tasks.cephadm:Deploying osd.2 on smithi076 with /dev/vg_nvme/lv_2... 2024-09-20T18:51:48.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T18:51:48.885 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:51:49.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[23648]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:49.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi076 to 7520M 2024-09-20T18:51:49.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[23648]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T18:51:49.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[23648]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T18:51:49.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[23648]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[30031]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi076 to 7520M 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[30031]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[30031]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-mon[30031]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:51:49.291 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1[37802]: 2024-09-20T18:51:49.156+0000 7f68638f9640 -1 osd.1 0 waiting for initial osdmap 2024-09-20T18:51:49.292 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:51:49 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1[37802]: 2024-09-20T18:51:49.164+0000 7f685f722640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:51:49.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:49 smithi194 ceph-mon[25770]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:49.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:49 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi076 to 7520M 2024-09-20T18:51:49.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:49 smithi194 ceph-mon[25770]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T18:51:49.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:49 smithi194 ceph-mon[25770]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T18:51:49.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:49 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:49.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:49 smithi194 ceph-mon[25770]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:51:50.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[23648]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:51:50.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[23648]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T18:51:50.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:50.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:50.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[23648]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:50.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[30031]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:51:50.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[30031]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T18:51:50.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:50.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:50.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:50 smithi076 ceph-mon[30031]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:50.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:50 smithi194 ceph-mon[25770]: from='osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:51:50.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:50 smithi194 ceph-mon[25770]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T18:51:50.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:50.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:50.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:50 smithi194 ceph-mon[25770]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T18:51:51.340 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:51:51.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2024-09-20T18:51:51.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:51:51.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:51:51.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:51.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[23648]: osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551] boot 2024-09-20T18:51:51.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[23648]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T18:51:51.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[30031]: osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551] boot 2024-09-20T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[30031]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:51 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:51.557 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:51:51.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:51 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:51:51.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:51 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:51:51.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:51 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:51.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:51 smithi194 ceph-mon[25770]: osd.1 [v2:172.21.15.76:6810/2678064551,v1:172.21.15.76:6811/2678064551] boot 2024-09-20T18:51:51.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:51 smithi194 ceph-mon[25770]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T18:51:51.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:51 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:51:52.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:52 smithi076 ceph-mon[23648]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T18:51:52.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:52 smithi076 ceph-mon[23648]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:52.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:52 smithi076 ceph-mon[30031]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T18:51:52.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:52 smithi076 ceph-mon[30031]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:52.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:52 smithi194 ceph-mon[25770]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T18:51:52.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:52 smithi194 ceph-mon[25770]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:53.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:53.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:53.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:53 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:53.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:53 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:53.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:53 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:53.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:53 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:53.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:51:53.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:51:53.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:53 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:54.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:54 smithi076 ceph-mon[23648]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:54.622 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:54 smithi076 ceph-mon[23648]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:54.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:54 smithi076 ceph-mon[30031]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:54.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:54 smithi076 ceph-mon[30031]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:54.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:54 smithi194 ceph-mon[25770]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:51:54.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:54 smithi194 ceph-mon[25770]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi076 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1536963405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c6e419-6bde-486c-93b9-9d370a617aea"}]: dispatch 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1536963405' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9c6e419-6bde-486c-93b9-9d370a617aea"}]': finished 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi076 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1536963405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c6e419-6bde-486c-93b9-9d370a617aea"}]: dispatch 2024-09-20T18:51:56.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1536963405' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9c6e419-6bde-486c-93b9-9d370a617aea"}]': finished 2024-09-20T18:51:56.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T18:51:56.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:56 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:51:56.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi076 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1536963405' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c6e419-6bde-486c-93b9-9d370a617aea"}]: dispatch 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1536963405' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9c6e419-6bde-486c-93b9-9d370a617aea"}]': finished 2024-09-20T18:51:56.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T18:51:56.597 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:56 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1957761638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:57.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1957761638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:57.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:51:57.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1957761638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:51:57.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:51:57.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:57.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:51:57.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:51:57.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:51:58.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:58 smithi076 ceph-mon[23648]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:58.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:58 smithi076 ceph-mon[23648]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:51:58.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:51:58 smithi076 ceph-mon[23648]: Cluster is now healthy 2024-09-20T18:51:58.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:58 smithi076 ceph-mon[30031]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:58.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:58 smithi076 ceph-mon[30031]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:51:58.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:51:58 smithi076 ceph-mon[30031]: Cluster is now healthy 2024-09-20T18:51:58.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:58 smithi194 ceph-mon[25770]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:51:58.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:58 smithi194 ceph-mon[25770]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:51:58.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:51:58 smithi194 ceph-mon[25770]: Cluster is now healthy 2024-09-20T18:52:00.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:00 smithi076 ceph-mon[23648]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:00.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:00 smithi076 ceph-mon[30031]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:00.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:00 smithi194 ceph-mon[25770]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:02.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:02 smithi076 ceph-mon[23648]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:02.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T18:52:02.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:02.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:02 smithi076 ceph-mon[30031]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:02.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T18:52:02.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:02.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:02 smithi194 ceph-mon[25770]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:02.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T18:52:02.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:03.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:03 smithi076 ceph-mon[23648]: Deploying daemon osd.2 on smithi076 2024-09-20T18:52:03.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:03 smithi076 ceph-mon[30031]: Deploying daemon osd.2 on smithi076 2024-09-20T18:52:03.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:03 smithi194 ceph-mon[25770]: Deploying daemon osd.2 on smithi076 2024-09-20T18:52:04.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:04 smithi076 ceph-mon[23648]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:04.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:04 smithi076 ceph-mon[30031]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:04.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:04 smithi194 ceph-mon[25770]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:06.398 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:06 smithi076 ceph-mon[23648]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:06.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:06 smithi076 ceph-mon[30031]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:06.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:06 smithi194 ceph-mon[25770]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:08.175 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.176 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:08.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:10.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:10 smithi076 ceph-mon[23648]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:10.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:10 smithi076 ceph-mon[30031]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:10.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:10 smithi194 ceph-mon[25770]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:11.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:11 smithi076 ceph-mon[23648]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T18:52:11.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:11 smithi076 ceph-mon[30031]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T18:52:11.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:11 smithi194 ceph-mon[25770]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T18:52:11.637 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 2 on host 'smithi076' 2024-09-20T18:52:12.258 DEBUG:teuthology.orchestra.run.smithi076:osd.2> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T18:52:12.261 INFO:tasks.cephadm:Deploying osd.3 on smithi076 with /dev/vg_nvme/lv_1... 2024-09-20T18:52:12.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T18:52:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:12.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:12.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:12.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:12.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:13.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi076 to 5013M 2024-09-20T18:52:13.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[23648]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:52:13.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[23648]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T18:52:13.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:13.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi076 to 5013M 2024-09-20T18:52:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[30031]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:52:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[30031]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T18:52:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:13 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:13.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:13 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi076 to 5013M 2024-09-20T18:52:13.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:13 smithi194 ceph-mon[25770]: from='osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:52:13.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:13 smithi194 ceph-mon[25770]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T18:52:13.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:13 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:13.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:13 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[23648]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[23648]: osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288] boot 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[23648]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[30031]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:14.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[30031]: osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288] boot 2024-09-20T18:52:14.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[30031]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T18:52:14.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:14 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:14 smithi194 ceph-mon[25770]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T18:52:14.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:14 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:14 smithi194 ceph-mon[25770]: osd.2 [v2:172.21.15.76:6818/1216770288,v1:172.21.15.76:6819/1216770288] boot 2024-09-20T18:52:14.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:14 smithi194 ceph-mon[25770]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T18:52:14.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:14 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:52:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:52:14.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2024-09-20T18:52:15.115 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[23648]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:15.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:52:15.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:52:15.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T18:52:15.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T18:52:15.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T18:52:15.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T18:52:15.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:15 smithi076 ceph-mon[30031]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:15.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:52:15.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:52:15.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T18:52:15.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T18:52:15.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T18:52:15.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T18:52:15.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:15 smithi194 ceph-mon[25770]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:16.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:16 smithi076 ceph-mon[23648]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T18:52:16.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:16 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T18:52:16.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:16 smithi076 ceph-mon[23648]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T18:52:16.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:16 smithi076 ceph-mon[30031]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T18:52:16.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:16 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T18:52:16.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:16 smithi076 ceph-mon[30031]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T18:52:16.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:16 smithi194 ceph-mon[25770]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T18:52:16.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:16 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T18:52:16.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:16 smithi194 ceph-mon[25770]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T18:52:16.772 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42525]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T18:52:16.772 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:16.772 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42525]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:16.772 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42525]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:17.040 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42614]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T18:52:17.040 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42614]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:17.040 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42614]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:17.040 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 18:52:16 smithi076 sudo[42614]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:17.352 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42779]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T18:52:17.352 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42779]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:17.352 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42779]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:17.352 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42779]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:17.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: pgmap v78: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi076 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:52:17.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: pgmap v78: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi076 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:17.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:17.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:52:17.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:52:17.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:52:17.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:17 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:17.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: pgmap v78: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi076 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:17.597 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:52:17.597 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:52:17.597 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:52:17.597 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:17 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:17.605 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42799]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T18:52:17.605 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42799]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:17.605 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42799]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:17.605 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42799]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:17.900 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T18:52:17.900 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:17.900 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42828]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:17.901 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42828]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:18.153 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42896]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T18:52:18.153 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42896]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:18.153 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42896]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:18.153 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 18:52:17 smithi076 sudo[42896]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:18.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42904]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T18:52:18.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42904]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:18.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42904]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:18.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42904]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 ceph-mon[25770]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 ceph-mon[25770]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 ceph-mon[25770]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29351]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:18.682 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29351]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:18.683 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29351]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:18.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:18 smithi076 ceph-mon[30031]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:52:18.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:18 smithi076 ceph-mon[30031]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T18:52:18.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:18 smithi076 ceph-mon[30031]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42921]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T18:52:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42921]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42921]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:18.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 sudo[42921]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:18.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 ceph-mon[23648]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:52:18.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 ceph-mon[23648]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T18:52:18.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:18 smithi076 ceph-mon[23648]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29354]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T18:52:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29354]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29354]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:18 smithi194 sudo[29354]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:19.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[42975]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T18:52:19.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[42975]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:19.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[42975]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:19.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[42975]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:19.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[43004]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T18:52:19.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[43004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T18:52:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[43004]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T18:52:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 sudo[43004]: pam_unix(sudo:session): session closed for user root 2024-09-20T18:52:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:19.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T18:52:19.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:19.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:19.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:19.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:19.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:19.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T18:52:19.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:19.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T18:52:19.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:19.847 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:52:19.847 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:52:19.847 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:52:20.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:20.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: pgmap v81: 1 pgs: 1 creating+activating; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:20.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4076204473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb383f16-34d4-482f-9a64-bc55b8b23d91"}]: dispatch 2024-09-20T18:52:20.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4076204473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb383f16-34d4-482f-9a64-bc55b8b23d91"}]': finished 2024-09-20T18:52:20.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T18:52:20.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:20.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:20.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:20.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:20.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:20.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:20.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: pgmap v81: 1 pgs: 1 creating+activating; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:20.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4076204473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb383f16-34d4-482f-9a64-bc55b8b23d91"}]: dispatch 2024-09-20T18:52:20.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4076204473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb383f16-34d4-482f-9a64-bc55b8b23d91"}]': finished 2024-09-20T18:52:20.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T18:52:20.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:20 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:20.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T18:52:20.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T18:52:20.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: pgmap v81: 1 pgs: 1 creating+activating; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4076204473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb383f16-34d4-482f-9a64-bc55b8b23d91"}]: dispatch 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4076204473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb383f16-34d4-482f-9a64-bc55b8b23d91"}]': finished 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T18:52:20.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:20 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:21.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:21 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1407917353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:52:21.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:21 smithi076 ceph-mon[23648]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:52:21.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:21 smithi076 ceph-mon[23648]: Cluster is now healthy 2024-09-20T18:52:21.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:21 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1407917353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:52:21.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:21 smithi076 ceph-mon[30031]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:52:21.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:21 smithi076 ceph-mon[30031]: Cluster is now healthy 2024-09-20T18:52:21.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:21 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1407917353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:52:21.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:21 smithi194 ceph-mon[25770]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:52:21.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:21 smithi194 ceph-mon[25770]: Cluster is now healthy 2024-09-20T18:52:22.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:22 smithi076 ceph-mon[30031]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:22.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:22 smithi076 ceph-mon[23648]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:22.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:22 smithi194 ceph-mon[25770]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:24.955 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:24 smithi076 ceph-mon[23648]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:24.956 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:24 smithi076 ceph-mon[30031]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:25.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:24 smithi194 ceph-mon[25770]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:26.885 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:26 smithi076 ceph-mon[23648]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:26.885 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:26 smithi076 ceph-mon[30031]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:27.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:26 smithi194 ceph-mon[25770]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:28.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[30031]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:28.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T18:52:28.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:28.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[23648]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:28.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T18:52:28.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:28.695 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.695 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.695 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:28.695 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:28 smithi194 ceph-mon[25770]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:28.695 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T18:52:28.696 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:29.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:29 smithi076 ceph-mon[23648]: Deploying daemon osd.3 on smithi076 2024-09-20T18:52:29.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:29 smithi076 ceph-mon[30031]: Deploying daemon osd.3 on smithi076 2024-09-20T18:52:29.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:29 smithi194 ceph-mon[25770]: Deploying daemon osd.3 on smithi076 2024-09-20T18:52:30.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:30 smithi194 ceph-mon[25770]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:30.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:30 smithi076 ceph-mon[23648]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:30.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:30 smithi076 ceph-mon[30031]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:33.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:32 smithi076 ceph-mon[23648]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:33.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:32 smithi076 ceph-mon[30031]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:33.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:32 smithi194 ceph-mon[25770]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:33.956 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:33.957 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:33.958 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:33 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:34.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:34.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:34.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:34.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:34.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:34.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:33 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:35.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:34 smithi076 ceph-mon[23648]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:35.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:34 smithi076 ceph-mon[30031]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:35.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:34 smithi194 ceph-mon[25770]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:36.344 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 3 on host 'smithi076' 2024-09-20T18:52:36.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:36.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:36.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[30031]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T18:52:36.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:36.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:36 smithi076 ceph-mon[23648]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T18:52:37.006 DEBUG:teuthology.orchestra.run.smithi076:osd.3> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T18:52:37.008 INFO:tasks.cephadm:Deploying osd.4 on smithi194 with /dev/vg_nvme/lv_4... 2024-09-20T18:52:37.008 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:52:37.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:37.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:37.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:37.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:37.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:36 smithi194 ceph-mon[25770]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T18:52:37.214 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:52:38.023 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:37 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi076 to 3760M 2024-09-20T18:52:38.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:37 smithi194 ceph-mon[25770]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T18:52:38.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:37 smithi194 ceph-mon[25770]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T18:52:38.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:37 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:38.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:37 smithi194 ceph-mon[25770]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:52:38.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi076 to 3760M 2024-09-20T18:52:38.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[23648]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T18:52:38.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[23648]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T18:52:38.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:38.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[23648]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:52:38.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi076 to 3760M 2024-09-20T18:52:38.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[30031]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T18:52:38.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[30031]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T18:52:38.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:38.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:37 smithi076 ceph-mon[30031]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-09-20T18:52:38.749 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T18:52:38.346+0000 7fd21b01f640 -1 osd.3 0 waiting for initial osdmap 2024-09-20T18:52:38.749 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T18:52:38.349+0000 7fd216e48640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:52:38.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:38 smithi194 ceph-mon[25770]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:38 smithi194 ceph-mon[25770]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:52:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:38 smithi194 ceph-mon[25770]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T18:52:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:39.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[30031]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:39.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[30031]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:52:39.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[30031]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T18:52:39.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:39.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:39.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[23648]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:39.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[23648]: from='osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-09-20T18:52:39.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[23648]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T18:52:39.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:39.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:38 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:39.584 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:52:39.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-09-20T18:52:39.836 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi076 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:40.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707] boot 2024-09-20T18:52:40.335 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: osdmap e25: 4 total, 4 up, 4 in 2024-09-20T18:52:40.335 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:40 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi076 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707] boot 2024-09-20T18:52:40.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: osdmap e25: 4 total, 4 up, 4 in 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi076 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: osd.3 [v2:172.21.15.76:6826/4109464707,v1:172.21.15.76:6827/4109464707] boot 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: osdmap e25: 4 total, 4 up, 4 in 2024-09-20T18:52:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:40 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:52:41.181 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:41 smithi194 ceph-mon[25770]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:41.181 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:41 smithi194 ceph-mon[25770]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T18:52:41.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:41 smithi076 ceph-mon[23648]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:41.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:41 smithi076 ceph-mon[23648]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T18:52:41.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:41 smithi076 ceph-mon[30031]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T18:52:41.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:41 smithi076 ceph-mon[30031]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T18:52:42.252 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:52:42.252 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:52:42.252 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:42.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:52:42.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:52:42.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:42.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:52:42.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:52:42.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:43.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:43 smithi076 ceph-mon[30031]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:52:43.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:43 smithi076 ceph-mon[30031]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:43 smithi076 ceph-mon[23648]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:52:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:43 smithi076 ceph-mon[23648]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:43 smithi194 ceph-mon[25770]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:52:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:43 smithi194 ceph-mon[25770]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:44.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/4246386112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]: dispatch 2024-09-20T18:52:44.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]: dispatch 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]': finished 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[23648]: osdmap e27: 5 total, 4 up, 5 in 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/4246386112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]: dispatch 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]: dispatch 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]': finished 2024-09-20T18:52:44.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[30031]: osdmap e27: 5 total, 4 up, 5 in 2024-09-20T18:52:44.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:52:44.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:44 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/4246386112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]: dispatch 2024-09-20T18:52:44.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:44 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]: dispatch 2024-09-20T18:52:44.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:44 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d60930d-7fdc-46b4-9204-cd4656919a5d"}]': finished 2024-09-20T18:52:44.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:44 smithi194 ceph-mon[25770]: osdmap e27: 5 total, 4 up, 5 in 2024-09-20T18:52:44.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:52:45.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:45 smithi194 ceph-mon[25770]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:45.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:45 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/972724794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:52:45.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:45 smithi076 ceph-mon[23648]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:45.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:45 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/972724794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:45 smithi076 ceph-mon[30031]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:45 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/972724794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:52:47.221 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:47 smithi194 ceph-mon[25770]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:47.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:47 smithi076 ceph-mon[23648]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:47.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:47 smithi076 ceph-mon[30031]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:48.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[23648]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:48.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:48 smithi076 ceph-mon[30031]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:48.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:48 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:48.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:48 smithi194 ceph-mon[25770]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:50 smithi076 ceph-mon[23648]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T18:52:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:51.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:50 smithi076 ceph-mon[30031]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:51.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T18:52:51.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:51.063 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:50 smithi194 ceph-mon[25770]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:51.063 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T18:52:51.063 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:51.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:51 smithi194 ceph-mon[25770]: Deploying daemon osd.4 on smithi194 2024-09-20T18:52:52.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:51 smithi076 ceph-mon[23648]: Deploying daemon osd.4 on smithi194 2024-09-20T18:52:52.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:51 smithi076 ceph-mon[30031]: Deploying daemon osd.4 on smithi194 2024-09-20T18:52:53.027 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:52 smithi194 ceph-mon[25770]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:53.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:52 smithi076 ceph-mon[23648]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:53.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:52 smithi076 ceph-mon[30031]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:55.024 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:54 smithi194 ceph-mon[25770]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:55.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:54 smithi076 ceph-mon[23648]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:55.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:54 smithi076 ceph-mon[30031]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:57.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:56 smithi076 ceph-mon[23648]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:57.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:56 smithi076 ceph-mon[30031]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:57.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:56 smithi194 ceph-mon[25770]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:57.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:57.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:57 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:57 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:52:58.909 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:58 smithi194 ceph-mon[25770]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:59.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:58 smithi076 ceph-mon[23648]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:59.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:58 smithi076 ceph-mon[30031]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:52:59.538 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 4 on host 'smithi194' 2024-09-20T18:53:00.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:00.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:00.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:00.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:52:59 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:00.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:00.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:00.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:00.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:52:59 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:00.165 DEBUG:teuthology.orchestra.run.smithi194:osd.4> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T18:53:00.168 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/vg_nvme/lv_3... 2024-09-20T18:53:00.168 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T18:53:00.396 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi194 to 16064M 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[23648]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[23648]: from='osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[23648]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi194 to 16064M 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[30031]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[30031]: from='osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T18:53:00.925 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:00 smithi076 ceph-mon[30031]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T18:53:01.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:00 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi194 to 16064M 2024-09-20T18:53:01.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:00 smithi194 ceph-mon[25770]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:01.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:00 smithi194 ceph-mon[25770]: from='osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T18:53:01.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:00 smithi194 ceph-mon[25770]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T18:53:01.903 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:01 smithi194 ceph-mon[25770]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T18:53:01.903 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:01 smithi194 ceph-mon[25770]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T18:53:01.903 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:01 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:01.903 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:01 smithi194 ceph-mon[25770]: from='osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:01.903 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:01 smithi194 ceph-mon[25770]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:02.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[23648]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T18:53:02.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[23648]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[23648]: from='osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[23648]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[30031]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[30031]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:02.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[30031]: from='osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:02.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:01 smithi076 ceph-mon[30031]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:02.512 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T18:53:02.205+0000 7f04bf422640 -1 osd.4 0 waiting for initial osdmap 2024-09-20T18:53:02.513 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T18:53:02.219+0000 7f04b7243640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:53:02.886 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:53:02.904 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-09-20T18:53:02.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[23648]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:02.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[23648]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:02.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[23648]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T18:53:02.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:02.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:02.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[23648]: from='osd.4 ' entity='osd.4' 2024-09-20T18:53:02.972 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[30031]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:02.972 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[30031]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:02.972 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[30031]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T18:53:02.972 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:02.972 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:02.972 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:02 smithi076 ceph-mon[30031]: from='osd.4 ' entity='osd.4' 2024-09-20T18:53:03.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-mon[25770]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:03.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-mon[25770]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:03.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-mon[25770]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T18:53:03.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:03.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:03.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:02 smithi194 ceph-mon[25770]: from='osd.4 ' entity='osd.4' 2024-09-20T18:53:03.101 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:53:04.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:53:04.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:53:04.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:04.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296] boot 2024-09-20T18:53:04.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: osdmap e30: 5 total, 5 up, 5 in 2024-09-20T18:53:04.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:04.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:03 smithi194 ceph-mon[25770]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T18:53:04.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:53:04.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296] boot 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: osdmap e30: 5 total, 5 up, 5 in 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[30031]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:53:04.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:53:04.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:04.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: osd.4 [v2:172.21.15.194:6800/2621521296,v1:172.21.15.194:6801/2621521296] boot 2024-09-20T18:53:04.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: osdmap e30: 5 total, 5 up, 5 in 2024-09-20T18:53:04.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:53:04.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:03 smithi076 ceph-mon[23648]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T18:53:05.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:04 smithi194 ceph-mon[25770]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:05.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:05.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:05.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:04 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:05.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:04 smithi194 ceph-mon[25770]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T18:53:05.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[23648]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:05.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:05.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:05.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:05.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[23648]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T18:53:05.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[30031]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T18:53:05.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:05.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:05.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:05.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:04 smithi076 ceph-mon[30031]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T18:53:06.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:05 smithi194 ceph-mon[25770]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:06.022 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:05 smithi194 ceph-mon[25770]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T18:53:06.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:05 smithi076 ceph-mon[30031]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:06.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:05 smithi076 ceph-mon[30031]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T18:53:06.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:05 smithi076 ceph-mon[23648]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:06.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:05 smithi076 ceph-mon[23648]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T18:53:07.047 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:06 smithi194 ceph-mon[25770]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:07.047 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:06 smithi194 ceph-mon[25770]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T18:53:07.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:06 smithi076 ceph-mon[30031]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:07.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:06 smithi076 ceph-mon[30031]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T18:53:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:06 smithi076 ceph-mon[23648]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:07.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:06 smithi076 ceph-mon[23648]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/4172962743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]: dispatch 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]: dispatch 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]': finished 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: osdmap e35: 6 total, 5 up, 6 in 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:08.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/1170249387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:08.097 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:07 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/4172962743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]: dispatch 2024-09-20T18:53:08.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]: dispatch 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]': finished 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: osdmap e35: 6 total, 5 up, 6 in 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/1170249387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/4172962743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]: dispatch 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]: dispatch 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781a2d6a-2292-4cd5-91bc-d2520e9016a2"}]': finished 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: osdmap e35: 6 total, 5 up, 6 in 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:08.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/1170249387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:08.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:07 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:09.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:08 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi194 2024-09-20T18:53:09.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:08 smithi194 ceph-mon[25770]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-20T18:53:09.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:08 smithi194 ceph-mon[25770]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-09-20T18:53:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:08 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi194 2024-09-20T18:53:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:08 smithi076 ceph-mon[23648]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-20T18:53:09.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:08 smithi076 ceph-mon[23648]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-09-20T18:53:09.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:08 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi194 2024-09-20T18:53:09.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:08 smithi076 ceph-mon[30031]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-20T18:53:09.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:08 smithi076 ceph-mon[30031]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-09-20T18:53:11.070 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:10 smithi194 ceph-mon[25770]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-20T18:53:11.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:10 smithi076 ceph-mon[23648]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-20T18:53:11.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:10 smithi076 ceph-mon[30031]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-20T18:53:13.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:12 smithi194 ceph-mon[25770]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-20T18:53:13.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:12 smithi076 ceph-mon[23648]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-20T18:53:13.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:12 smithi076 ceph-mon[30031]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-20T18:53:15.053 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:14 smithi194 ceph-mon[25770]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:15.054 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:14 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T18:53:15.054 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:14 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:15.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:14 smithi076 ceph-mon[23648]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:15.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:14 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T18:53:15.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:14 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:15.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:14 smithi076 ceph-mon[30031]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:15.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:14 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T18:53:15.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:14 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:15.962 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:15 smithi194 ceph-mon[25770]: Deploying daemon osd.5 on smithi194 2024-09-20T18:53:16.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:15 smithi076 ceph-mon[30031]: Deploying daemon osd.5 on smithi194 2024-09-20T18:53:16.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:15 smithi076 ceph-mon[23648]: Deploying daemon osd.5 on smithi194 2024-09-20T18:53:16.977 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:16 smithi194 ceph-mon[25770]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:17.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:16 smithi076 ceph-mon[23648]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:17.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:16 smithi076 ceph-mon[30031]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:18.754 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:18 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.754 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:18 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.754 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:18 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.754 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:18 smithi194 ceph-mon[25770]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[23648]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:18.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:18.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:18 smithi076 ceph-mon[30031]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:19.692 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:19.692 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.692 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:19.692 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:19.692 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.692 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:19 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:19.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:19.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:19.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:19.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:19 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:20.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:20 smithi194 ceph-mon[25770]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:20.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:20 smithi076 ceph-mon[23648]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:20.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:20 smithi076 ceph-mon[30031]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:21.961 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-09-20T18:53:22.628 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T18:53:22.630 INFO:tasks.cephadm:Deploying osd.6 on smithi194 with /dev/vg_nvme/lv_2... 2024-09-20T18:53:22.631 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T18:53:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:53:22.981 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi194 to 8032M 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T18:53:22.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:22 smithi194 ceph-mon[25770]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T18:53:23.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:23.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:23.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:23.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi194 to 8032M 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[30031]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi194 to 8032M 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T18:53:23.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:22 smithi076 ceph-mon[23648]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T18:53:24.164 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-mon[25770]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T18:53:24.164 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-mon[25770]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T18:53:24.164 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:24.164 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-mon[25770]: from='osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:24.164 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-mon[25770]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:24.164 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T18:53:23.962+0000 7fd8c26bf640 -1 osd.5 0 waiting for initial osdmap 2024-09-20T18:53:24.164 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 18:53:23 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T18:53:23.966+0000 7fd8be4e8640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:53:24.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[23648]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T18:53:24.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[23648]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T18:53:24.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:24.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[23648]: from='osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:24.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[23648]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:24.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[30031]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T18:53:24.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[30031]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T18:53:24.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:24.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[30031]: from='osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:24.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:23 smithi076 ceph-mon[30031]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:25.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[23648]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:25.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[23648]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:25.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[23648]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T18:53:25.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:25.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:25.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[30031]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:25.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[30031]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:25.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[30031]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T18:53:25.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:25.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:25.302 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:24 smithi194 ceph-mon[25770]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T18:53:25.302 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:24 smithi194 ceph-mon[25770]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:25.302 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:24 smithi194 ceph-mon[25770]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T18:53:25.303 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:25.303 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:25.306 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:53:25.324 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-09-20T18:53:25.517 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:53:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:53:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:53:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[23648]: osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537] boot 2024-09-20T18:53:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[23648]: osdmap e38: 6 total, 6 up, 6 in 2024-09-20T18:53:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:26.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[23648]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T18:53:26.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:53:26.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:53:26.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[30031]: osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537] boot 2024-09-20T18:53:26.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[30031]: osdmap e38: 6 total, 6 up, 6 in 2024-09-20T18:53:26.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:26.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:25 smithi076 ceph-mon[30031]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T18:53:26.309 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:25 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:53:26.309 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:25 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:53:26.309 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:25 smithi194 ceph-mon[25770]: osd.5 [v2:172.21.15.194:6808/3127477537,v1:172.21.15.194:6809/3127477537] boot 2024-09-20T18:53:26.309 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:25 smithi194 ceph-mon[25770]: osdmap e38: 6 total, 6 up, 6 in 2024-09-20T18:53:26.309 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:25 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:53:26.309 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:25 smithi194 ceph-mon[25770]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T18:53:27.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:26 smithi076 ceph-mon[30031]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:27.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:26 smithi076 ceph-mon[30031]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T18:53:27.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:26 smithi076 ceph-mon[23648]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:27.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:26 smithi076 ceph-mon[23648]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T18:53:27.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:26 smithi194 ceph-mon[25770]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:27.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:26 smithi194 ceph-mon[25770]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T18:53:28.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T18:53:28.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:28.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi194 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi194 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:28.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:28.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:28.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:28 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi194 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:28.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:28 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:29.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:29 smithi194 ceph-mon[25770]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:53:29.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:29 smithi194 ceph-mon[25770]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T18:53:29.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:29 smithi076 ceph-mon[23648]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:53:29.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:29 smithi076 ceph-mon[23648]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T18:53:29.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:29 smithi076 ceph-mon[30031]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:53:29.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:29 smithi076 ceph-mon[30031]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T18:53:30.321 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:30 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/244543626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]: dispatch 2024-09-20T18:53:30.322 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:30 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]: dispatch 2024-09-20T18:53:30.322 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:30 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]': finished 2024-09-20T18:53:30.322 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:30 smithi194 ceph-mon[25770]: osdmap e43: 7 total, 6 up, 7 in 2024-09-20T18:53:30.322 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:30 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:30.322 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:30 smithi194 ceph-mon[25770]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/244543626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]: dispatch 2024-09-20T18:53:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]: dispatch 2024-09-20T18:53:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]': finished 2024-09-20T18:53:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[23648]: osdmap e43: 7 total, 6 up, 7 in 2024-09-20T18:53:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[23648]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:30.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/244543626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]: dispatch 2024-09-20T18:53:30.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]: dispatch 2024-09-20T18:53:30.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13d285c9-2ca1-4e31-9833-d2e07ba67ad9"}]': finished 2024-09-20T18:53:30.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[30031]: osdmap e43: 7 total, 6 up, 7 in 2024-09-20T18:53:30.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:30.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:30 smithi076 ceph-mon[30031]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:31.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:31 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/278822805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:31.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:31 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/278822805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:31.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:31 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/278822805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:32.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:33 smithi194 ceph-mon[25770]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:53:33.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:33 smithi194 ceph-mon[25770]: Cluster is now healthy 2024-09-20T18:53:33.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:33 smithi076 ceph-mon[23648]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:53:33.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:33 smithi076 ceph-mon[23648]: Cluster is now healthy 2024-09-20T18:53:33.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:33 smithi076 ceph-mon[30031]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:53:33.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:33 smithi076 ceph-mon[30031]: Cluster is now healthy 2024-09-20T18:53:34.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:34 smithi076 ceph-mon[23648]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:34.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:34 smithi076 ceph-mon[30031]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:34.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:34 smithi194 ceph-mon[25770]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:36.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:36 smithi076 ceph-mon[23648]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:36.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:36 smithi076 ceph-mon[30031]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:36.814 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:36 smithi194 ceph-mon[25770]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:37.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[23648]: Deploying daemon osd.6 on smithi194 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[30031]: Deploying daemon osd.6 on smithi194 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:37 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:37 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T18:53:37.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:37 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:37.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:37 smithi194 ceph-mon[25770]: Deploying daemon osd.6 on smithi194 2024-09-20T18:53:37.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:37 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:37 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:37.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:37 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:38.708 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:38 smithi194 ceph-mon[25770]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:38.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:38 smithi076 ceph-mon[23648]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:38.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:38 smithi076 ceph-mon[30031]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:40.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:40 smithi076 ceph-mon[23648]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:40.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:40 smithi076 ceph-mon[30031]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:40.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:40 smithi194 ceph-mon[25770]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:42.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:42.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:42.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:42.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:42 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:42.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:42.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:42.813 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:42.813 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.813 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:42.813 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:42 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.369 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 6 on host 'smithi194' 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:44.689 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:44 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:53:44.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:44.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:44.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:44.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:44 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:45.001 DEBUG:teuthology.orchestra.run.smithi194:osd.6> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T18:53:45.004 INFO:tasks.cephadm:Deploying osd.7 on smithi194 with /dev/vg_nvme/lv_1... 2024-09-20T18:53:45.004 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T18:53:45.237 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:53:45.548 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:45 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi194 to 5354M 2024-09-20T18:53:45.548 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:45 smithi194 ceph-mon[25770]: from='osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T18:53:45.548 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:45 smithi194 ceph-mon[25770]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T18:53:45.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:45 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi194 to 5354M 2024-09-20T18:53:45.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:45 smithi076 ceph-mon[23648]: from='osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T18:53:45.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:45 smithi076 ceph-mon[23648]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T18:53:45.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:45 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi194 to 5354M 2024-09-20T18:53:45.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:45 smithi076 ceph-mon[30031]: from='osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T18:53:45.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:45 smithi076 ceph-mon[30031]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T18:53:46.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-mon[25770]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:46.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-mon[25770]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T18:53:46.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-mon[25770]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T18:53:46.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:46.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-mon[25770]: from='osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:46.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-mon[25770]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:47.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[23648]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:47.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[23648]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T18:53:47.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[23648]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[23648]: from='osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[23648]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[30031]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[30031]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[30031]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[30031]: from='osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:47.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:46 smithi076 ceph-mon[30031]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:53:47.346 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T18:53:46.979+0000 7f58b2a7e640 -1 osd.6 0 waiting for initial osdmap 2024-09-20T18:53:47.346 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 18:53:46 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T18:53:46.987+0000 7f58aa89f640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:53:47.730 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-20T18:53:47.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-09-20T18:53:47.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:47.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T18:53:47.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:47.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:47.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: from='osd.6 ' entity='osd.6' 2024-09-20T18:53:47.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638] boot 2024-09-20T18:53:47.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: osdmap e46: 7 total, 7 up, 7 in 2024-09-20T18:53:47.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:47 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:47.944 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: from='osd.6 ' entity='osd.6' 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638] boot 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: osdmap e46: 7 total, 7 up, 7 in 2024-09-20T18:53:48.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: from='osd.6 ' entity='osd.6' 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: osd.6 [v2:172.21.15.194:6816/3710031638,v1:172.21.15.194:6817/3710031638] boot 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: osdmap e46: 7 total, 7 up, 7 in 2024-09-20T18:53:48.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:47 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:53:48.650 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:48 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:53:48.650 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:48 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:53:48.650 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:48 smithi194 ceph-mon[25770]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:48.650 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:48 smithi194 ceph-mon[25770]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T18:53:49.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:53:49.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:53:49.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[23648]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:49.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[23648]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T18:53:49.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:53:49.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:53:49.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[30031]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T18:53:49.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:48 smithi076 ceph-mon[30031]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T18:53:50.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: pgmap v154: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:50.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T18:53:50.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:50.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:50.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:50.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:50.847 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:50 smithi194 ceph-mon[25770]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T18:53:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: pgmap v154: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T18:53:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:51.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[23648]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: pgmap v154: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:51.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:50 smithi076 ceph-mon[30031]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T18:53:52.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T18:53:52.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:52.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi194 2024-09-20T18:53:52.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:52.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:52.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:52.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: pgmap v159: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:52.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T18:53:52.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:52.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi194 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: pgmap v159: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:52.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:52 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi194 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: pgmap v159: 1 pgs: 1 remapped; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-20T18:53:52.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:52 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:53 smithi076 ceph-mon[23648]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:53:53.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:53 smithi076 ceph-mon[30031]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:53:53.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:53 smithi194 ceph-mon[25770]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T18:53:54.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/2750305901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]: dispatch 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]: dispatch 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]': finished 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: osdmap e51: 8 total, 7 up, 8 in 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:53:54.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:54 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/700575037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:54.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/2750305901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]: dispatch 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]: dispatch 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]': finished 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: osdmap e51: 8 total, 7 up, 8 in 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:53:54.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/700575037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/2750305901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]: dispatch 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]: dispatch 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4af47219-dca1-47a6-9141-4f23fc64e636"}]': finished 2024-09-20T18:53:54.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: osdmap e51: 8 total, 7 up, 8 in 2024-09-20T18:53:54.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:53:54.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:54 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/700575037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T18:53:55.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:55 smithi076 ceph-mon[23648]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:53:55.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:55 smithi076 ceph-mon[23648]: Cluster is now healthy 2024-09-20T18:53:55.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:55 smithi076 ceph-mon[30031]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:53:55.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:55 smithi076 ceph-mon[30031]: Cluster is now healthy 2024-09-20T18:53:55.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:55 smithi194 ceph-mon[25770]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T18:53:55.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:55 smithi194 ceph-mon[25770]: Cluster is now healthy 2024-09-20T18:53:56.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:56 smithi076 ceph-mon[23648]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-20T18:53:56.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:56 smithi076 ceph-mon[30031]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-20T18:53:56.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:56 smithi194 ceph-mon[25770]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-20T18:53:58.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:58 smithi194 ceph-mon[25770]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-20T18:53:58.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:58 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:58.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:58 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:58.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:53:58 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:59.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[23648]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-20T18:53:59.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:59.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:59.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:59.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[30031]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-20T18:53:59.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:59.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:53:59.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:53:58 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:01.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:00 smithi076 ceph-mon[23648]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T18:54:01.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:00 smithi076 ceph-mon[30031]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T18:54:01.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:00 smithi194 ceph-mon[25770]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T18:54:01.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:01 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T18:54:01.837 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:01 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:01.837 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:01 smithi194 ceph-mon[25770]: Deploying daemon osd.7 on smithi194 2024-09-20T18:54:01.940 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:01 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T18:54:01.940 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:01 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:01.941 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:01 smithi076 ceph-mon[23648]: Deploying daemon osd.7 on smithi194 2024-09-20T18:54:01.941 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:01 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T18:54:01.941 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:01 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:01.941 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:01 smithi076 ceph-mon[30031]: Deploying daemon osd.7 on smithi194 2024-09-20T18:54:02.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:02 smithi076 ceph-mon[23648]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T18:54:02.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:02 smithi076 ceph-mon[30031]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T18:54:02.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:02 smithi194 ceph-mon[25770]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T18:54:04.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:04 smithi194 ceph-mon[25770]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:04.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:04 smithi076 ceph-mon[23648]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:04.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:04 smithi076 ceph-mon[30031]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:06.952 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:06 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:07.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:07.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:06 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:08.616 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 7 on host 'smithi194' 2024-09-20T18:54:08.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:08.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:08.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:08.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:08.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:08.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:08.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:08 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:08 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.288 DEBUG:teuthology.orchestra.run.smithi194:osd.7> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T18:54:09.291 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T18:54:09.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T18:54:09.496 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: Adjusting osd_memory_target on smithi194 to 4016M 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T18:54:09.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[23648]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: Adjusting osd_memory_target on smithi194 to 4016M 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T18:54:09.975 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:09 smithi076 ceph-mon[30031]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T18:54:10.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:10.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: Adjusting osd_memory_target on smithi194 to 4016M 2024-09-20T18:54:10.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:10.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:10.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:10.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:10.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T18:54:10.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:09 smithi194 ceph-mon[25770]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T18:54:10.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-mon[25770]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:10.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-mon[25770]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T18:54:10.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-mon[25770]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T18:54:10.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:10.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-mon[25770]: from='osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:54:10.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-mon[25770]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:54:10.846 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T18:54:10.628+0000 7f17c293d640 -1 osd.7 0 waiting for initial osdmap 2024-09-20T18:54:10.846 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 18:54:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T18:54:10.637+0000 7f17be766640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T18:54:10.900 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[23648]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:10.900 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[23648]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[23648]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[23648]: from='osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[23648]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[30031]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[30031]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[30031]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[30031]: from='osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:54:10.901 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:10 smithi076 ceph-mon[30031]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-20T18:54:11.426 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:54:11.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:54:11.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T18:54:11.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1094515821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221] boot 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: osdmap e54: 8 total, 8 up, 8 in 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1094515821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221] boot 2024-09-20T18:54:11.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: osdmap e54: 8 total, 8 up, 8 in 2024-09-20T18:54:11.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:11 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:12.065 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726858427,"num_in_osds":8,"osd_in_since":1726858433,"num_remapped_pgs":0} 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1094515821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: osd.7 [v2:172.21.15.194:6824/2797123221,v1:172.21.15.194:6825/2797123221] boot 2024-09-20T18:54:12.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: osdmap e54: 8 total, 8 up, 8 in 2024-09-20T18:54:12.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:11 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:13.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: purged_snaps scrub starts 2024-09-20T18:54:13.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: purged_snaps scrub ok 2024-09-20T18:54:13.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:13.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi194 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[23648]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: purged_snaps scrub starts 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: purged_snaps scrub ok 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi194 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:12 smithi076 ceph-mon[30031]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T18:54:13.066 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T18:54:13.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: purged_snaps scrub starts 2024-09-20T18:54:13.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: purged_snaps scrub ok 2024-09-20T18:54:13.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T18:54:13.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi194 2024-09-20T18:54:13.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:13.096 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:12 smithi194 ceph-mon[25770]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T18:54:13.267 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:54:14.859 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:14 smithi194 ceph-mon[25770]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:14.860 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:14 smithi194 ceph-mon[25770]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T18:54:14.894 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:14 smithi076 ceph-mon[23648]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:14.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:14 smithi076 ceph-mon[23648]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T18:54:14.895 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:14 smithi076 ceph-mon[30031]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:14.895 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:14 smithi076 ceph-mon[30031]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T18:54:14.981 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:54:15.611 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1726858451,"num_in_osds":8,"osd_in_since":1726858433,"num_remapped_pgs":0} 2024-09-20T18:54:15.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T18:54:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:54:15.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:15 smithi194 ceph-mon[25770]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T18:54:15.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:15 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2077705083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T18:54:15.940 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:15 smithi076 ceph-mon[23648]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T18:54:15.940 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:15 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2077705083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T18:54:15.940 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:15 smithi076 ceph-mon[30031]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T18:54:15.940 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:15 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2077705083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T18:54:16.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:16 smithi194 ceph-mon[25770]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:16.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:16 smithi194 ceph-mon[25770]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T18:54:16.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:16 smithi076 ceph-mon[23648]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:16.931 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:16 smithi076 ceph-mon[23648]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T18:54:16.931 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:16 smithi076 ceph-mon[30031]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:16.931 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:16 smithi076 ceph-mon[30031]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T18:54:17.600 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:54:17.600 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":58,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","created":"2024-09-20T18:49:18.760609+0000","modified":"2024-09-20T18:54:15.632826+0000","last_up_change":"2024-09-20T18:54:11.619890+0000","last_in_change":"2024-09-20T18:53:53.692431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T18:52:14.243494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"802eb93b-c4da-44ae-ab87-912b6094a939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6803","nonce":2779901970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6805","nonce":2779901970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6809","nonce":2779901970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6807","nonce":2779901970}]},"public_addr":"172.21.15.76:6803/2779901970","cluster_addr":"172.21.15.76:6805/2779901970","heartbeat_back_addr":"172.21.15.76:6809/2779901970","heartbeat_front_addr":"172.21.15.76:6807/2779901970","state":["exists","up"]},{"osd":1,"uuid":"deafe02c-4749-4ca9-bc3b-5537831c731c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6811","nonce":2678064551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6813","nonce":2678064551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6817","nonce":2678064551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6815","nonce":2678064551}]},"public_addr":"172.21.15.76:6811/2678064551","cluster_addr":"172.21.15.76:6813/2678064551","heartbeat_back_addr":"172.21.15.76:6817/2678064551","heartbeat_front_addr":"172.21.15.76:6815/2678064551","state":["exists","up"]},{"osd":2,"uuid":"d9c6e419-6bde-486c-93b9-9d370a617aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6819","nonce":1216770288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6821","nonce":1216770288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6825","nonce":1216770288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6823","nonce":1216770288}]},"public_addr":"172.21.15.76:6819/1216770288","cluster_addr":"172.21.15.76:6821/1216770288","heartbeat_back_addr":"172.21.15.76:6825/1216770288","heartbeat_front_addr":"172.21.15.76:6823/1216770288","state":["exists","up"]},{"osd":3,"uuid":"eb383f16-34d4-482f-9a64-bc55b8b23d91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6827","nonce":4109464707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6829","nonce":4109464707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6833","nonce":4109464707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6831","nonce":4109464707}]},"public_addr":"172.21.15.76:6827/4109464707","cluster_addr":"172.21.15.76:6829/4109464707","heartbeat_back_addr":"172.21.15.76:6833/4109464707","heartbeat_front_addr":"172.21.15.76:6831/4109464707","state":["exists","up"]},{"osd":4,"uuid":"1d60930d-7fdc-46b4-9204-cd4656919a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621521296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621521296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621521296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621521296}]},"public_addr":"172.21.15.194:6801/2621521296","cluster_addr":"172.21.15.194:6803/2621521296","heartbeat_back_addr":"172.21.15.194:6807/2621521296","heartbeat_front_addr":"172.21.15.194:6805/2621521296","state":["exists","up"]},{"osd":5,"uuid":"781a2d6a-2292-4cd5-91bc-d2520e9016a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6809","nonce":3127477537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6811","nonce":3127477537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6815","nonce":3127477537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6813","nonce":3127477537}]},"public_addr":"172.21.15.194:6809/3127477537","cluster_addr":"172.21.15.194:6811/3127477537","heartbeat_back_addr":"172.21.15.194:6815/3127477537","heartbeat_front_addr":"172.21.15.194:6813/3127477537","state":["exists","up"]},{"osd":6,"uuid":"13d285c9-2ca1-4e31-9833-d2e07ba67ad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6817","nonce":3710031638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6819","nonce":3710031638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6823","nonce":3710031638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6821","nonce":3710031638}]},"public_addr":"172.21.15.194:6817/3710031638","cluster_addr":"172.21.15.194:6819/3710031638","heartbeat_back_addr":"172.21.15.194:6823/3710031638","heartbeat_front_addr":"172.21.15.194:6821/3710031638","state":["exists","up"]},{"osd":7,"uuid":"4af47219-dca1-47a6-9141-4f23fc64e636","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6825","nonce":2797123221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6827","nonce":2797123221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6831","nonce":2797123221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6829","nonce":2797123221}]},"public_addr":"172.21.15.194:6825/2797123221","cluster_addr":"172.21.15.194:6827/2797123221","heartbeat_back_addr":"172.21.15.194:6831/2797123221","heartbeat_front_addr":"172.21.15.194:6829/2797123221","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:51:25.337535+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:51:48.413403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:52:12.045777+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:52:37.571359+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:01.123041+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:23.478735+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:45.849737+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:54:10.104633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/2786028271":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6800/2786028271":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/2932735846":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/785531962":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6801/1767800311":"2024-09-21T18:49:41.970878+0000","172.21.15.76:6800/1767800311":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/2497469942":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/1658935611":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/3296258683":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/3293279753":"2024-09-21T18:49:41.970878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T18:54:17.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:17 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3556332723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:54:17.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:17 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3556332723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:54:18.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:17 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3556332723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:54:18.343 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T18:52:14.243494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-20T18:54:18.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T18:54:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:54:18.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[23648]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-20T18:54:18.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:18.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:18.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:18.973 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[30031]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-20T18:54:18.974 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:18.974 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:18.974 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:18 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:18 smithi194 ceph-mon[25770]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-20T18:54:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:18 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:18 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:19.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:18 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:20.419 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-09-20T18:54:20.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:20 smithi076 ceph-mon[23648]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T18:54:20.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:20 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2500455150' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T18:54:21.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:20 smithi076 ceph-mon[30031]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T18:54:21.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:20 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2500455150' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T18:54:21.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:20 smithi194 ceph-mon[25770]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T18:54:21.095 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:20 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2500455150' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T18:54:21.163 INFO:tasks.cephadm:Adding prometheus.a on smithi194 2024-09-20T18:54:21.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch apply prometheus '1;smithi194=a' 2024-09-20T18:54:21.375 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:54:23.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:22 smithi076 ceph-mon[23648]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-20T18:54:23.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:22 smithi076 ceph-mon[30031]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-20T18:54:23.094 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:22 smithi194 ceph-mon[25770]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-20T18:54:23.130 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled prometheus update... 2024-09-20T18:54:23.743 DEBUG:teuthology.orchestra.run.smithi194:prometheus.a> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@prometheus.a.service 2024-09-20T18:54:23.745 INFO:tasks.cephadm:Adding node-exporter.a on smithi076 2024-09-20T18:54:23.745 INFO:tasks.cephadm:Adding node-exporter.b on smithi194 2024-09-20T18:54:23.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch apply node-exporter '2;smithi076=a;smithi194=b' 2024-09-20T18:54:23.987 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:54:24.441 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:24.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: Saving service prometheus spec with placement smithi194=a;count:1 2024-09-20T18:54:24.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:24.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:24.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:24.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:24.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:24 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: Saving service prometheus spec with placement smithi194=a;count:1 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:24.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: Saving service prometheus spec with placement smithi194=a;count:1 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:24.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:24 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:25.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:25 smithi194 ceph-mon[25770]: Deploying daemon prometheus.a on smithi194 2024-09-20T18:54:25.494 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:25 smithi194 ceph-mon[25770]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T18:54:25.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:25 smithi076 ceph-mon[30031]: Deploying daemon prometheus.a on smithi194 2024-09-20T18:54:25.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:25 smithi076 ceph-mon[30031]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T18:54:25.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:25 smithi076 ceph-mon[23648]: Deploying daemon prometheus.a on smithi194 2024-09-20T18:54:25.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:25 smithi076 ceph-mon[23648]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T18:54:26.165 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled node-exporter update... 2024-09-20T18:54:27.261 DEBUG:teuthology.orchestra.run.smithi076:node-exporter.a> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.a.service 2024-09-20T18:54:27.264 DEBUG:teuthology.orchestra.run.smithi194:node-exporter.b> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.b.service 2024-09-20T18:54:27.267 INFO:tasks.cephadm:Adding alertmanager.a on smithi076 2024-09-20T18:54:27.267 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch apply alertmanager '1;smithi076=a' 2024-09-20T18:54:27.503 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:54:27.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[23648]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T18:54:27.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[23648]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi076=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:27.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[23648]: Saving service node-exporter spec with placement smithi076=a;smithi194=b;count:2 2024-09-20T18:54:27.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:27.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[30031]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T18:54:27.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[30031]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi076=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:27.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[30031]: Saving service node-exporter spec with placement smithi076=a;smithi194=b;count:2 2024-09-20T18:54:27.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:27 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:27 smithi194 ceph-mon[25770]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T18:54:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:27 smithi194 ceph-mon[25770]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi076=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:27 smithi194 ceph-mon[25770]: Saving service node-exporter spec with placement smithi076=a;smithi194=b;count:2 2024-09-20T18:54:27.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:27 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:29.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:29 smithi076 ceph-mon[23648]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T18:54:29.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:29 smithi076 ceph-mon[30031]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T18:54:29.576 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:29 smithi194 ceph-mon[25770]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T18:54:31.249 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled alertmanager update... 2024-09-20T18:54:31.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:31 smithi194 ceph-mon[25770]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:31.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:31 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:31 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:31 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[30031]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:31.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[23648]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:31.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:31.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:31 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:32.443 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:32 smithi194 ceph-mon[25770]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi076=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:32.444 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:32 smithi194 ceph-mon[25770]: Saving service alertmanager spec with placement smithi076=a;count:1 2024-09-20T18:54:32.444 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:32 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:32.444 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:32 smithi194 ceph-mon[25770]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:32.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[30031]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi076=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:32.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[30031]: Saving service alertmanager spec with placement smithi076=a;count:1 2024-09-20T18:54:32.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:32.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[30031]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[23648]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi076=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[23648]: Saving service alertmanager spec with placement smithi076=a;count:1 2024-09-20T18:54:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:32 smithi076 ceph-mon[23648]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:34.758 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:34 smithi194 ceph-mon[25770]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:34.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:34 smithi076 ceph-mon[23648]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:34.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:34 smithi076 ceph-mon[30031]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:35.594 DEBUG:teuthology.orchestra.run.smithi076:alertmanager.a> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@alertmanager.a.service 2024-09-20T18:54:35.599 INFO:tasks.cephadm:Adding grafana.a on smithi194 2024-09-20T18:54:35.599 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph orch apply grafana '1;smithi194=a' 2024-09-20T18:54:35.844 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:35 smithi194 podman[44745]: 2024-09-20 18:54:35.807493513 +0000 UTC m=+0.144992646 container create 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:54:36.223 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:54:36.345 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 podman[44745]: 2024-09-20 18:54:36.068375147 +0000 UTC m=+0.405874294 container init 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:54:36.345 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 podman[44745]: 2024-09-20 18:54:36.071453162 +0000 UTC m=+0.408952296 container start 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:54:36.345 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.095Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T18:54:36.345 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.095Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T18:54:36.345 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.095Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi194 (none))" 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.095Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.095Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.096Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.096Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.099Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.099Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.100Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.100Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.093µs 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.100Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.100Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.100Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.83µs wal_replay_duration=180.043µs wbl_replay_duration=88ns total_replay_duration=220.837µs 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.101Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.101Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T18:54:36.346 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.101Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T18:54:36.347 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.120Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.604939ms db_storage=740ns remote_storage=1.161µs web_handler=188ns query_engine=547ns scrape=442.448µs scrape_sd=71.222µs notify=890ns notify_sd=2.226µs rules=18.810007ms tracing=6.846µs 2024-09-20T18:54:36.347 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.120Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T18:54:36.347 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:54:36.120Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T18:54:36.347 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 bash[44745]: 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 2024-09-20T18:54:36.347 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:54:36 smithi194 systemd[1]: Started Ceph prometheus.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:54:36.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-mon[25770]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:36.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:36 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T18:54:36.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[23648]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:36.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T18:54:36.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[30031]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:36.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' 2024-09-20T18:54:36.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:36 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T18:54:37.456 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: ignoring --setuser ceph since I am not root 2024-09-20T18:54:37.456 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: ignoring --setgroup ceph since I am not root 2024-09-20T18:54:37.456 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-mgr[27954]: -- 172.21.15.194:0/1555424238 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55730267ad00 con 0x55730264e800 2024-09-20T18:54:37.456 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:37.384+0000 7f0b4fafd040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T18:54:37.457 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:37.439+0000 7f0b4fafd040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T18:54:37.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ignoring --setuser ceph since I am not root 2024-09-20T18:54:37.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ignoring --setgroup ceph since I am not root 2024-09-20T18:54:37.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:37.369+0000 7fed58448040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T18:54:37.539 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:37.424+0000 7fed58448040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T18:54:37.718 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:37.601+0000 7f0b4fafd040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T18:54:37.718 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:37.669+0000 7f0b4fafd040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T18:54:37.965 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:37.574+0000 7fed58448040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T18:54:37.966 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:37.643+0000 7fed58448040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T18:54:37.966 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:37.807+0000 7fed58448040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:54:37.966 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:37.948+0000 7fed58448040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T18:54:37.992 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:37.816+0000 7f0b4fafd040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:54:37.992 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:37 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:37.975+0000 7f0b4fafd040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T18:54:38.263 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:38 smithi194 ceph-mon[25770]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T18:54:38.263 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:38 smithi194 ceph-mon[25770]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T18:54:38.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-mon[23648]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T18:54:38.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-mon[23648]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T18:54:38.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-mon[30031]: from='mgr.14150 172.21.15.76:0/143085128' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T18:54:38.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-mon[30031]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T18:54:38.510 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:38.254+0000 7fed58448040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T18:54:38.510 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:38.314+0000 7fed58448040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T18:54:38.566 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:38 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:38.300+0000 7f0b4fafd040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T18:54:38.567 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:38 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:38.366+0000 7f0b4fafd040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T18:54:38.567 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:38 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:38.550+0000 7f0b4fafd040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T18:54:38.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:38 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:38.492+0000 7fed58448040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T18:54:39.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:39 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:39.345+0000 7fed58448040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T18:54:39.789 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:39 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:39.410+0000 7fed58448040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T18:54:39.844 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:39 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:39.420+0000 7f0b4fafd040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T18:54:39.844 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:39 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:39.486+0000 7f0b4fafd040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T18:54:40.387 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.115+0000 7fed58448040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T18:54:40.387 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.228+0000 7fed58448040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T18:54:40.463 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.192+0000 7f0b4fafd040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T18:54:40.463 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.307+0000 7f0b4fafd040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T18:54:40.660 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.370+0000 7fed58448040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:54:40.660 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.432+0000 7fed58448040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T18:54:40.661 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.577+0000 7fed58448040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T18:54:40.661 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.644+0000 7fed58448040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T18:54:40.735 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.447+0000 7f0b4fafd040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T18:54:40.735 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.507+0000 7f0b4fafd040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T18:54:40.735 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.652+0000 7f0b4fafd040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T18:54:40.735 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.718+0000 7f0b4fafd040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T18:54:40.942 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.703+0000 7fed58448040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T18:54:40.942 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.859+0000 7fed58448040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T18:54:40.942 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.925+0000 7fed58448040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T18:54:41.014 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.778+0000 7f0b4fafd040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T18:54:41.014 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.932+0000 7f0b4fafd040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T18:54:41.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:40.992+0000 7fed58448040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T18:54:41.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:41.073+0000 7fed58448040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T18:54:41.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:41.132+0000 7fed58448040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T18:54:41.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:41.190+0000 7fed58448040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T18:54:41.280 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:40 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:40.997+0000 7f0b4fafd040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T18:54:41.280 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:41 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:41.064+0000 7f0b4fafd040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T18:54:41.280 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:41 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:41.146+0000 7f0b4fafd040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T18:54:41.281 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:41 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:41.205+0000 7f0b4fafd040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T18:54:41.281 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:41 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:41.264+0000 7f0b4fafd040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:41 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:41.961+0000 7fed58448040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:54:42] ENGINE Bus STARTING 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: CherryPy Checker: 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: The Application mounted at '' has an empty config. 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:54:42] ENGINE Serving on http://:::9283 2024-09-20T18:54:42.289 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:54:42] ENGINE Bus STARTED 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: Active manager daemon y restarted 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: Activating manager daemon y 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: mgrmap e17: y(active, starting, since 0.00803963s), standbys: x 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T18:54:42.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: Manager daemon y is now available 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:42.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]': finished 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: Active manager daemon y restarted 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: Activating manager daemon y 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: mgrmap e17: y(active, starting, since 0.00803963s), standbys: x 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T18:54:42.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: Manager daemon y is now available 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:42.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:42.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]': finished 2024-09-20T18:54:42.344 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:42.044+0000 7f0b4fafd040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: [20/Sep/2024:18:54:42] ENGINE Bus STARTING 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: CherryPy Checker: 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: The Application mounted at '' has an empty config. 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: [20/Sep/2024:18:54:42] ENGINE Serving on http://:::9283 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x[27950]: [20/Sep/2024:18:54:42] ENGINE Bus STARTED 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: Active manager daemon y restarted 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: Activating manager daemon y 2024-09-20T18:54:42.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: mgrmap e17: y(active, starting, since 0.00803963s), standbys: x 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T18:54:42.346 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: Manager daemon y is now available 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-09-20T18:54:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]': finished 2024-09-20T18:54:43.002 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled grafana update... 2024-09-20T18:54:43.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]': finished 2024-09-20T18:54:43.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: Standby manager daemon x restarted 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: Standby manager daemon x started 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T18:54:43.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: mgrmap e18: y(active, since 1.01373s), standbys: x 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]': finished 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: Standby manager daemon x restarted 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: Standby manager daemon x started 2024-09-20T18:54:43.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T18:54:43.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T18:54:43.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T18:54:43.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T18:54:43.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T18:54:43.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: mgrmap e18: y(active, since 1.01373s), standbys: x 2024-09-20T18:54:43.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:43.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]': finished 2024-09-20T18:54:43.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:54:43.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:43.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: Updating smithi076:/etc/ceph/ceph.conf 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: Standby manager daemon x restarted 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: Standby manager daemon x started 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.? 172.21.15.194:0/1770486631' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: mgrmap e18: y(active, since 1.01373s), standbys: x 2024-09-20T18:54:43.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:43.762 DEBUG:teuthology.orchestra.run.smithi194:grafana.a> sudo journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@grafana.a.service 2024-09-20T18:54:43.764 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T18:54:43.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T18:54:43.956 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:54:44.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:43.663+0000 7fecb0877640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:18:54:43] ENGINE Error in HTTPServer.serve 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: Traceback (most recent call last): 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._connections.run(self.expiration_interval) 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._run(expiration_interval) 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: s = self.context.wrap_socket( 2024-09-20T18:54:44.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: return self.sslsocket_class._create( 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self.do_handshake() 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: self._sslobj.do_handshake() 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:54:44.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:43 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: 2024-09-20T18:54:44.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:54:44.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:54:44.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:54:44.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Saving service grafana spec with placement smithi194=a;count:1 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.345 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Saving service grafana spec with placement smithi194=a;count:1 2024-09-20T18:54:44.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T18:54:44.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Saving service grafana spec with placement smithi194=a;count:1 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Updating smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: Updating smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:44.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: Deploying daemon alertmanager.a on smithi076 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:54:43] ENGINE Bus STARTING 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:54:43] ENGINE Serving on http://172.21.15.76:8765 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:54:43] ENGINE Serving on https://172.21.15.76:7151 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:54:43] ENGINE Bus STARTED 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: [20/Sep/2024:18:54:43] ENGINE Error in HTTPServer.serve 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout: Traceback (most recent call last): 2024-09-20T18:54:45.344 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._connections.run(self.expiration_interval) 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._run(expiration_interval) 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: s = self.context.wrap_socket( 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: return self.sslsocket_class._create( 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: self.do_handshake() 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: self._sslobj.do_handshake() 2024-09-20T18:54:45.345 INFO:journalctl@ceph.mon.a.smithi076.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[23648]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: Deploying daemon alertmanager.a on smithi076 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: [20/Sep/2024:18:54:43] ENGINE Bus STARTING 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: [20/Sep/2024:18:54:43] ENGINE Serving on http://172.21.15.76:8765 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: [20/Sep/2024:18:54:43] ENGINE Serving on https://172.21.15.76:7151 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: [20/Sep/2024:18:54:43] ENGINE Bus STARTED 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: [20/Sep/2024:18:54:43] ENGINE Error in HTTPServer.serve 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout: Traceback (most recent call last): 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout: self._connections.run(self.expiration_interval) 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:54:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout: self._run(expiration_interval) 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: s = self.context.wrap_socket( 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: return self.sslsocket_class._create( 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: self.do_handshake() 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: self._sslobj.do_handshake() 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:45 smithi076 ceph-mon[30031]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: Deploying daemon alertmanager.a on smithi076 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: [20/Sep/2024:18:54:43] ENGINE Bus STARTING 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: [20/Sep/2024:18:54:43] ENGINE Serving on http://172.21.15.76:8765 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: [20/Sep/2024:18:54:43] ENGINE Serving on https://172.21.15.76:7151 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: [20/Sep/2024:18:54:43] ENGINE Bus STARTED 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: [20/Sep/2024:18:54:43] ENGINE Error in HTTPServer.serve 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: Traceback (most recent call last): 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: self._connections.run(self.expiration_interval) 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: self._run(expiration_interval) 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T18:54:45.348 INFO:journalctl@ceph.mon.b.smithi194.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: s = self.context.wrap_socket( 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: return self.sslsocket_class._create( 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: self.do_handshake() 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: self._sslobj.do_handshake() 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:45.349 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:45 smithi194 ceph-mon[25770]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T18:54:47.328 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:47 smithi194 ceph-mon[25770]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:47.328 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:47 smithi194 ceph-mon[25770]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T18:54:47.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:47 smithi076 ceph-mon[23648]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:47.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:47 smithi076 ceph-mon[23648]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T18:54:47.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:47 smithi076 ceph-mon[30031]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:47.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:47 smithi076 ceph-mon[30031]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T18:54:49.260 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-09-20T18:54:49.260 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQD5xO1mB4MyDhAAFK4laTMKqIvgvJWTPovJDw== 2024-09-20T18:54:49.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:49 smithi194 ceph-mon[25770]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:49.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:49 smithi076 ceph-mon[23648]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:49.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:49 smithi076 ceph-mon[30031]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:54:50.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3278476779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:50.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-mon[23648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:50.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-mon[23648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T18:54:50.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3278476779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:50.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-mon[30031]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:50.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-mon[30031]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T18:54:50.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:50 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3278476779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:50.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:50 smithi194 ceph-mon[25770]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:50.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:50 smithi194 ceph-mon[25770]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T18:54:50.641 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:54:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:54:50] "GET /metrics HTTP/1.1" 200 23673 "" "Prometheus/2.43.0" 2024-09-20T18:54:51.453 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-20T18:54:51.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T18:54:51.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T18:54:51.496 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T18:54:51.524 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:51 smithi194 ceph-mon[25770]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T18:54:51.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:51 smithi076 ceph-mon[23648]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T18:54:51.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:51 smithi076 ceph-mon[30031]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T18:54:51.703 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T18:54:53.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:53 smithi076 ceph-mon[23648]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T18:54:53.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:53 smithi076 ceph-mon[30031]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T18:54:53.465 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:53 smithi076 systemd[1]: Starting Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:54:53.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:53 smithi194 ceph-mon[25770]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T18:54:54.039 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:53 smithi076 podman[50018]: 2024-09-20 18:54:53.622707367 +0000 UTC m=+0.015730137 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-20T18:54:54.039 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:53 smithi076 podman[50018]: 2024-09-20 18:54:53.823182061 +0000 UTC m=+0.216204827 volume create 0f15fafec5a824fac564f46104d2823e79e57df01e181f1caf11119f5293c684 2024-09-20T18:54:54.039 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:53 smithi076 podman[50018]: 2024-09-20 18:54:53.940120607 +0000 UTC m=+0.333143374 container create 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 podman[50018]: 2024-09-20 18:54:54.267964715 +0000 UTC m=+0.660987489 container init 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 podman[50018]: 2024-09-20 18:54:54.27110767 +0000 UTC m=+0.664130450 container start 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.283Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.284Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.285Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.76 port=9094 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.286Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.309Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.309Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.312Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:54.312Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T18:54:54.539 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 bash[50018]: 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec 2024-09-20T18:54:54.540 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:54 smithi076 systemd[1]: Started Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:54:55.331 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:55 smithi194 ceph-mon[25770]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T18:54:55.331 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:55 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.331 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:55 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.331 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:55 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.331 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:55 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.412 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2024-09-20T18:54:55.412 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD/xO1miUthFxAAc5nsweWbnD0Zbob1pp1eFQ== 2024-09-20T18:54:55.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[23648]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T18:54:55.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[30031]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T18:54:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:55 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:56.308 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-20T18:54:56.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T18:54:56.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T18:54:56.389 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-20T18:54:56.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:56 smithi194 ceph-mon[25770]: Deploying daemon node-exporter.a on smithi076 2024-09-20T18:54:56.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:56 smithi194 ceph-mon[25770]: from='client.? 172.21.15.194:0/45016194' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:56.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:56 smithi194 ceph-mon[25770]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:56.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:56 smithi194 ceph-mon[25770]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T18:54:56.538 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:54:56.286Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000088519s 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[23648]: Deploying daemon node-exporter.a on smithi076 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[23648]: from='client.? 172.21.15.194:0/45016194' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[23648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[23648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[30031]: Deploying daemon node-exporter.a on smithi076 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[30031]: from='client.? 172.21.15.194:0/45016194' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:56.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[30031]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T18:54:56.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:56 smithi076 ceph-mon[30031]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T18:54:57.166 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:54:57.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:57 smithi076 ceph-mon[30031]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:54:57.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:57.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:54:57.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:57 smithi076 ceph-mon[23648]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:54:57.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:57.318 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:54:57.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:57 smithi194 ceph-mon[25770]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:54:57.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:54:57.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:54:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:54:59.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:59 smithi076 ceph-mon[23648]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:54:59.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:54:59 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/938601872' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T18:54:59.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:59 smithi076 ceph-mon[30031]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:54:59.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:54:59 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/938601872' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T18:54:59.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:59 smithi194 ceph-mon[25770]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:54:59.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:54:59 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/938601872' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T18:54:59.720 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:54:59 smithi076 bash[50628]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T18:55:00.788 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:55:00] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-20T18:55:01.039 INFO:teuthology.orchestra.run.smithi076.stdout:[{"version":22,"timestamp":"2024-09-20T18:54:42.014389+0000","name":"","changes":[{"name":"osd/host:smithi194/osd_memory_target","previous_value":"4211117158"}]}] 2024-09-20T18:55:01.039 INFO:tasks.ceph_manager:config epoch is 22 2024-09-20T18:55:01.040 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T18:55:01.040 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T18:55:01.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T18:55:01.269 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:01.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:01 smithi076 ceph-mon[23648]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:55:01.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:01 smithi076 ceph-mon[30031]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:55:01.594 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:01 smithi194 ceph-mon[25770]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T18:55:01.835 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:01 smithi076 bash[50628]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T18:55:02.191 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:01 smithi076 bash[50628]: Writing manifest to image destination 2024-09-20T18:55:02.777 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 podman[50628]: 2024-09-20 18:55:02.471945378 +0000 UTC m=+3.586845699 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T18:55:02.778 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 podman[50628]: 2024-09-20 18:55:02.653605901 +0000 UTC m=+3.768506231 container create b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 podman[50628]: 2024-09-20 18:55:02.880491431 +0000 UTC m=+3.995391758 container init b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 podman[50628]: 2024-09-20 18:55:02.883431115 +0000 UTC m=+3.998331437 container start b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.887Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.887Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T18:55:03.286 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T18:55:03.287 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T18:55:03.288 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T18:55:03.289 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.888Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a[50963]: ts=2024-09-20T18:55:02.889Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 bash[50628]: b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 2024-09-20T18:55:03.290 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 18:55:02 smithi076 systemd[1]: Started Ceph node-exporter.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:55:03.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[30031]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:03.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[23648]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:03.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:03 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.573 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:03 smithi194 ceph-mon[25770]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:03.573 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:03 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.573 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:03 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.573 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:03 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:03.836 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:03 smithi194 systemd[1]: Starting Ceph node-exporter.b for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:55:04.094 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:04 smithi194 bash[46809]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T18:55:04.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:04 smithi076 ceph-mon[23648]: Deploying daemon node-exporter.b on smithi194 2024-09-20T18:55:04.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:04 smithi076 ceph-mon[30031]: Deploying daemon node-exporter.b on smithi194 2024-09-20T18:55:04.430 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:55:04 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:55:04.289Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002616634s 2024-09-20T18:55:04.455 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:55:04.593 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:04 smithi194 ceph-mon[25770]: Deploying daemon node-exporter.b on smithi194 2024-09-20T18:55:04.594 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:04 smithi194 bash[46809]: Getting image source signatures 2024-09-20T18:55:04.594 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:04 smithi194 bash[46809]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T18:55:04.594 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:04 smithi194 bash[46809]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T18:55:04.594 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:04 smithi194 bash[46809]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T18:55:05.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:05 smithi076 ceph-mon[23648]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:05.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:05 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3532106088' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T18:55:05.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:05 smithi076 ceph-mon[30031]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:05.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:05 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3532106088' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T18:55:05.593 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:05 smithi194 ceph-mon[25770]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:05.593 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:05 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3532106088' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T18:55:06.060 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":20,"flags":0,"active_gid":14445,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":3898755307},{"type":"v1","addr":"172.21.15.76:6801","nonce":3898755307}]},"active_addr":"172.21.15.76:6801/3898755307","active_change":"2024-09-20T18:54:41.963842+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24344,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.76:8443/","prometheus":"http://172.21.15.76:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2562967750}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2331166204}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":1953411175}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":542469493}]}]} 2024-09-20T18:55:06.064 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T18:55:06.064 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T18:55:06.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T18:55:06.093 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:05 smithi194 bash[46809]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T18:55:06.093 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:05 smithi194 bash[46809]: Writing manifest to image destination 2024-09-20T18:55:06.543 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 podman[46809]: 2024-09-20 18:55:06.211915808 +0000 UTC m=+2.288347434 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T18:55:06.543 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 podman[46809]: 2024-09-20 18:55:06.310145632 +0000 UTC m=+2.386577253 container create ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T18:55:06.543 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 podman[46809]: 2024-09-20 18:55:06.528570323 +0000 UTC m=+2.605001945 container init ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T18:55:06.717 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 podman[46809]: 2024-09-20 18:55:06.531699579 +0000 UTC m=+2.608131215 container start ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.535Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.535Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.536Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.536Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.536Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.536Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T18:55:06.844 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T18:55:06.845 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T18:55:06.846 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T18:55:06.847 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b[46896]: ts=2024-09-20T18:55:06.537Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 bash[46809]: ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf 2024-09-20T18:55:06.848 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 18:55:06 smithi194 systemd[1]: Started Ceph node-exporter.b for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:55:07.049 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:06 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:55:06] ENGINE Bus STOPPING 2024-09-20T18:55:07.049 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:06 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:55:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T18:55:07.049 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:06 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:55:06] ENGINE Bus STOPPED 2024-09-20T18:55:07.049 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:06 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:55:06] ENGINE Bus STARTING 2024-09-20T18:55:07.049 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:55:07] ENGINE Serving on http://:::9283 2024-09-20T18:55:07.049 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:55:07] ENGINE Bus STARTED 2024-09-20T18:55:07.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:07.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.564 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T18:55:07.564 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:07.564 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:08.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:08 smithi076 ceph-mon[23648]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T18:55:08.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:08 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T18:55:08.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:08 smithi076 ceph-mon[23648]: Deploying daemon grafana.a on smithi194 2024-09-20T18:55:08.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:08 smithi076 ceph-mon[30031]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T18:55:08.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:08 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T18:55:08.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:08 smithi076 ceph-mon[30031]: Deploying daemon grafana.a on smithi194 2024-09-20T18:55:08.584 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:08 smithi194 ceph-mon[25770]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T18:55:08.584 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:08 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T18:55:08.584 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:08 smithi194 ceph-mon[25770]: Deploying daemon grafana.a on smithi194 2024-09-20T18:55:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:55:09.213 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":59,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","created":"2024-09-20T18:49:18.760609+0000","modified":"2024-09-20T18:54:41.963708+0000","last_up_change":"2024-09-20T18:54:11.619890+0000","last_in_change":"2024-09-20T18:53:53.692431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T18:52:14.243494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"802eb93b-c4da-44ae-ab87-912b6094a939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6803","nonce":2779901970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6805","nonce":2779901970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6809","nonce":2779901970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6807","nonce":2779901970}]},"public_addr":"172.21.15.76:6803/2779901970","cluster_addr":"172.21.15.76:6805/2779901970","heartbeat_back_addr":"172.21.15.76:6809/2779901970","heartbeat_front_addr":"172.21.15.76:6807/2779901970","state":["exists","up"]},{"osd":1,"uuid":"deafe02c-4749-4ca9-bc3b-5537831c731c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6811","nonce":2678064551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6813","nonce":2678064551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6817","nonce":2678064551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6815","nonce":2678064551}]},"public_addr":"172.21.15.76:6811/2678064551","cluster_addr":"172.21.15.76:6813/2678064551","heartbeat_back_addr":"172.21.15.76:6817/2678064551","heartbeat_front_addr":"172.21.15.76:6815/2678064551","state":["exists","up"]},{"osd":2,"uuid":"d9c6e419-6bde-486c-93b9-9d370a617aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6819","nonce":1216770288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6821","nonce":1216770288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6825","nonce":1216770288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6823","nonce":1216770288}]},"public_addr":"172.21.15.76:6819/1216770288","cluster_addr":"172.21.15.76:6821/1216770288","heartbeat_back_addr":"172.21.15.76:6825/1216770288","heartbeat_front_addr":"172.21.15.76:6823/1216770288","state":["exists","up"]},{"osd":3,"uuid":"eb383f16-34d4-482f-9a64-bc55b8b23d91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6827","nonce":4109464707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6829","nonce":4109464707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6833","nonce":4109464707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6831","nonce":4109464707}]},"public_addr":"172.21.15.76:6827/4109464707","cluster_addr":"172.21.15.76:6829/4109464707","heartbeat_back_addr":"172.21.15.76:6833/4109464707","heartbeat_front_addr":"172.21.15.76:6831/4109464707","state":["exists","up"]},{"osd":4,"uuid":"1d60930d-7fdc-46b4-9204-cd4656919a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621521296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621521296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621521296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621521296}]},"public_addr":"172.21.15.194:6801/2621521296","cluster_addr":"172.21.15.194:6803/2621521296","heartbeat_back_addr":"172.21.15.194:6807/2621521296","heartbeat_front_addr":"172.21.15.194:6805/2621521296","state":["exists","up"]},{"osd":5,"uuid":"781a2d6a-2292-4cd5-91bc-d2520e9016a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6809","nonce":3127477537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6811","nonce":3127477537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6815","nonce":3127477537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6813","nonce":3127477537}]},"public_addr":"172.21.15.194:6809/3127477537","cluster_addr":"172.21.15.194:6811/3127477537","heartbeat_back_addr":"172.21.15.194:6815/3127477537","heartbeat_front_addr":"172.21.15.194:6813/3127477537","state":["exists","up"]},{"osd":6,"uuid":"13d285c9-2ca1-4e31-9833-d2e07ba67ad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6817","nonce":3710031638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6819","nonce":3710031638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6823","nonce":3710031638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6821","nonce":3710031638}]},"public_addr":"172.21.15.194:6817/3710031638","cluster_addr":"172.21.15.194:6819/3710031638","heartbeat_back_addr":"172.21.15.194:6823/3710031638","heartbeat_front_addr":"172.21.15.194:6821/3710031638","state":["exists","up"]},{"osd":7,"uuid":"4af47219-dca1-47a6-9141-4f23fc64e636","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6825","nonce":2797123221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6827","nonce":2797123221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6831","nonce":2797123221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6829","nonce":2797123221}]},"public_addr":"172.21.15.194:6825/2797123221","cluster_addr":"172.21.15.194:6827/2797123221","heartbeat_back_addr":"172.21.15.194:6831/2797123221","heartbeat_front_addr":"172.21.15.194:6829/2797123221","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:51:25.337535+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:51:48.413403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:52:12.045777+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:52:37.571359+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:01.123041+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:23.478735+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:45.849737+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:54:10.104633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/1200527284":"2024-09-21T18:54:41.963677+0000","172.21.15.76:6801/999442551":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/3293279753":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/1528313282":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/3296258683":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/1658935611":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6800/1767800311":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/4123750440":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/2497469942":"2024-09-21T18:49:41.970878+0000","172.21.15.76:6801/1767800311":"2024-09-21T18:49:41.970878+0000","172.21.15.76:6800/999442551":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/785531962":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/2932735846":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6800/2786028271":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6801/2786028271":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/2133093190":"2024-09-21T18:54:41.963677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T18:55:09.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:09 smithi076 ceph-mon[23648]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:09.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:09 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1399422875' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:55:09.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:09 smithi076 ceph-mon[30031]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:09.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:09 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1399422875' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:55:09.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:09 smithi194 ceph-mon[25770]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:09.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:09 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1399422875' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:55:10.210 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T18:55:10.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T18:55:10.440 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:10 smithi076 ceph-mon[23648]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:10.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:10 smithi076 ceph-mon[30031]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:10.593 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:10 smithi194 ceph-mon[25770]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:10.760 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:55:10] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-20T18:55:11.357 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:13.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:13 smithi076 ceph-mon[23648]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:13.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:13 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:13.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:13 smithi076 ceph-mon[30031]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:13.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:13 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:13.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:13 smithi194 ceph-mon[25770]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:13.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:13 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:55:13.935 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":59,"fsid":"eefb3510-7780-11ef-baf5-efdc52797490","created":"2024-09-20T18:49:18.760609+0000","modified":"2024-09-20T18:54:41.963708+0000","last_up_change":"2024-09-20T18:54:11.619890+0000","last_in_change":"2024-09-20T18:53:53.692431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T18:52:14.243494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"802eb93b-c4da-44ae-ab87-912b6094a939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6803","nonce":2779901970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6805","nonce":2779901970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6809","nonce":2779901970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":2779901970},{"type":"v1","addr":"172.21.15.76:6807","nonce":2779901970}]},"public_addr":"172.21.15.76:6803/2779901970","cluster_addr":"172.21.15.76:6805/2779901970","heartbeat_back_addr":"172.21.15.76:6809/2779901970","heartbeat_front_addr":"172.21.15.76:6807/2779901970","state":["exists","up"]},{"osd":1,"uuid":"deafe02c-4749-4ca9-bc3b-5537831c731c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6811","nonce":2678064551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6813","nonce":2678064551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6817","nonce":2678064551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2678064551},{"type":"v1","addr":"172.21.15.76:6815","nonce":2678064551}]},"public_addr":"172.21.15.76:6811/2678064551","cluster_addr":"172.21.15.76:6813/2678064551","heartbeat_back_addr":"172.21.15.76:6817/2678064551","heartbeat_front_addr":"172.21.15.76:6815/2678064551","state":["exists","up"]},{"osd":2,"uuid":"d9c6e419-6bde-486c-93b9-9d370a617aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6819","nonce":1216770288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6821","nonce":1216770288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6825","nonce":1216770288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1216770288},{"type":"v1","addr":"172.21.15.76:6823","nonce":1216770288}]},"public_addr":"172.21.15.76:6819/1216770288","cluster_addr":"172.21.15.76:6821/1216770288","heartbeat_back_addr":"172.21.15.76:6825/1216770288","heartbeat_front_addr":"172.21.15.76:6823/1216770288","state":["exists","up"]},{"osd":3,"uuid":"eb383f16-34d4-482f-9a64-bc55b8b23d91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6827","nonce":4109464707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6829","nonce":4109464707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6833","nonce":4109464707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":4109464707},{"type":"v1","addr":"172.21.15.76:6831","nonce":4109464707}]},"public_addr":"172.21.15.76:6827/4109464707","cluster_addr":"172.21.15.76:6829/4109464707","heartbeat_back_addr":"172.21.15.76:6833/4109464707","heartbeat_front_addr":"172.21.15.76:6831/4109464707","state":["exists","up"]},{"osd":4,"uuid":"1d60930d-7fdc-46b4-9204-cd4656919a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621521296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621521296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621521296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621521296},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621521296}]},"public_addr":"172.21.15.194:6801/2621521296","cluster_addr":"172.21.15.194:6803/2621521296","heartbeat_back_addr":"172.21.15.194:6807/2621521296","heartbeat_front_addr":"172.21.15.194:6805/2621521296","state":["exists","up"]},{"osd":5,"uuid":"781a2d6a-2292-4cd5-91bc-d2520e9016a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6809","nonce":3127477537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6811","nonce":3127477537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6815","nonce":3127477537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3127477537},{"type":"v1","addr":"172.21.15.194:6813","nonce":3127477537}]},"public_addr":"172.21.15.194:6809/3127477537","cluster_addr":"172.21.15.194:6811/3127477537","heartbeat_back_addr":"172.21.15.194:6815/3127477537","heartbeat_front_addr":"172.21.15.194:6813/3127477537","state":["exists","up"]},{"osd":6,"uuid":"13d285c9-2ca1-4e31-9833-d2e07ba67ad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6817","nonce":3710031638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6819","nonce":3710031638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6823","nonce":3710031638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3710031638},{"type":"v1","addr":"172.21.15.194:6821","nonce":3710031638}]},"public_addr":"172.21.15.194:6817/3710031638","cluster_addr":"172.21.15.194:6819/3710031638","heartbeat_back_addr":"172.21.15.194:6823/3710031638","heartbeat_front_addr":"172.21.15.194:6821/3710031638","state":["exists","up"]},{"osd":7,"uuid":"4af47219-dca1-47a6-9141-4f23fc64e636","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6825","nonce":2797123221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6827","nonce":2797123221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6831","nonce":2797123221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2797123221},{"type":"v1","addr":"172.21.15.194:6829","nonce":2797123221}]},"public_addr":"172.21.15.194:6825/2797123221","cluster_addr":"172.21.15.194:6827/2797123221","heartbeat_back_addr":"172.21.15.194:6831/2797123221","heartbeat_front_addr":"172.21.15.194:6829/2797123221","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:51:25.337535+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:51:48.413403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:52:12.045777+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:52:37.571359+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:01.123041+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:23.478735+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:53:45.849737+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T18:54:10.104633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:0/1200527284":"2024-09-21T18:54:41.963677+0000","172.21.15.76:6801/999442551":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/3293279753":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/1528313282":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/3296258683":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/1658935611":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6800/1767800311":"2024-09-21T18:49:41.970878+0000","172.21.15.76:0/4123750440":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/2497469942":"2024-09-21T18:49:41.970878+0000","172.21.15.76:6801/1767800311":"2024-09-21T18:49:41.970878+0000","172.21.15.76:6800/999442551":"2024-09-21T18:54:41.963677+0000","172.21.15.76:0/785531962":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/2932735846":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6800/2786028271":"2024-09-21T18:50:03.937086+0000","172.21.15.76:6801/2786028271":"2024-09-21T18:50:03.937086+0000","172.21.15.76:0/2133093190":"2024-09-21T18:54:41.963677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T18:55:14.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:14 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3313865855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:55:14.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:14 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3313865855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:55:14.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:14 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3313865855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T18:55:14.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T18:55:14.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T18:55:14.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T18:55:14.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T18:55:14.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T18:55:14.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T18:55:14.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T18:55:14.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T18:55:15.160 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.160 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.161 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.161 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.166 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.170 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.171 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.180 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:15.306 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:15 smithi076 ceph-mon[23648]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:15.306 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:15 smithi076 ceph-mon[30031]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:15.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:15 smithi194 ceph-mon[25770]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:17.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:17 smithi076 ceph-mon[23648]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:17.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:17 smithi076 ceph-mon[30031]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:17.312 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:17 smithi194 ceph-mon[25770]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:18.002 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:17 smithi194 systemd[1]: Starting Ceph grafana.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:55:18.324 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:17 smithi194 podman[47721]: 2024-09-20 18:55:17.888862898 +0000 UTC m=+0.014956518 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-20T18:55:18.325 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 podman[47721]: 2024-09-20 18:55:18.033000069 +0000 UTC m=+0.159093676 container create 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a, vendor=Red Hat, Inc., description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-type=git, version=8.5, io.openshift.expose-services=, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public) 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 podman[47721]: 2024-09-20 18:55:18.310334946 +0000 UTC m=+0.436428568 container init 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vendor=Red Hat, Inc., io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base rhel8, distribution-scope=public, architecture=x86_64, com.redhat.component=ubi8-container) 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 podman[47721]: 2024-09-20 18:55:18.313130299 +0000 UTC m=+0.439223910 container start 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, maintainer=Paul Cuzner , version=8.5, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=ubi8-container, name=ubi8, vcs-type=git, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.buildah.version=1.32.0) 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 bash[47721]: 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 systemd[1]: Started Ceph grafana.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475381286Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475717557Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475737057Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475744388Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-20T18:55:18.576 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475750493Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475756539Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475764724Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475772415Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475780181Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475787787Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475793216Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475798506Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475804017Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=settings t=2024-09-20T18:55:18.475809495Z level=info msg="App mode production" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=sqlstore t=2024-09-20T18:55:18.47587534Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=sqlstore t=2024-09-20T18:55:18.475894479Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.476791126Z level=info msg="Starting DB migrations" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.476876091Z level=info msg="Executing migration" id="create migration_log table" 2024-09-20T18:55:18.577 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.477421481Z level=info msg="Executing migration" id="create user table" 2024-09-20T18:55:18.578 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.477888457Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-20T18:55:18.578 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.478695261Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-20T18:55:18.578 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.47952429Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-20T18:55:18.578 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.480344851Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-20T18:55:18.578 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.481279744Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-20T18:55:18.578 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.482249346Z level=info msg="Executing migration" id="create user table v2" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.483126883Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.484056515Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.48496342Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.485804085Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.486669895Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.487518444Z level=info msg="Executing migration" id="Update user table charset" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.488365883Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.488848599Z level=info msg="Executing migration" id="Add missing user data" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.489717171Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.490605525Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.491487551Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.492323865Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.493535606Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.494411818Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-20T18:55:18.579 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.495251353Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.49567755Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.496529342Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.499463194Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.502254207Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.50501181Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.507777271Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.508228394Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.51098844Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.511645385Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.514082951Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.516804541Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.517215957Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.520127694Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-20T18:55:18.580 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.520641397Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.522903645Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.526395429Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.529135764Z level=info msg="Executing migration" id="create star table" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.531981091Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.535497177Z level=info msg="Executing migration" id="create org table v1" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.537324202Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.541061717Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.542507711Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.545263146Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.547858723Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.550547265Z level=info msg="Executing migration" id="Update org table charset" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.553266195Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.555897255Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.558344172Z level=info msg="Executing migration" id="create dashboard table" 2024-09-20T18:55:18.581 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.561117661Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-20T18:55:18.827 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.563700227Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-20T18:55:18.827 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.566472907Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-20T18:55:18.827 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.568947539Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.571710875Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.574559061Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.577172068Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.579856713Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.582352705Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.583328305Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.583998132Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.585040393Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.585678104Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.586652228Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-20T18:55:18.828 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.587340233Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-20T18:55:18.829 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.588218664Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-20T18:55:18.829 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.589163911Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-20T18:55:18.829 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.589921749Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-20T18:55:18.829 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.590732078Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-20T18:55:18.829 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.591595743Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.592480593Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.593335108Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.594200439Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.595017366Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.595909641Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.596755485Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.597606734Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.598589279Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.599370116Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.600336339Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.601082149Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-20T18:55:18.830 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.601928745Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.602828638Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.603703239Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.604589443Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.605476929Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.606323765Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.607215788Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.6080686Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.608923295Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.609773279Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.610656812Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.611509186Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.612401695Z level=info msg="Executing migration" id="create data_source table" 2024-09-20T18:55:18.831 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.615267183Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.618182369Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.620973692Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.623712996Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.626707103Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.629422152Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.632125986Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.632569537Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.63354118Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.634263433Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.635108624Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.635960812Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-20T18:55:18.832 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.636890562Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.637680571Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.638520659Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.639437652Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.640432318Z level=info msg="Executing migration" id="Add uid column" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.641298654Z level=info msg="Executing migration" id="Update uid value" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.642192416Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.64301244Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.643901891Z level=info msg="Executing migration" id="create api_key table" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.646614236Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.649490935Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-20T18:55:18.833 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.652216552Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.655089212Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.657766841Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.658356926Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.661155366Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.662213836Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.664052046Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.666758233Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.669709879Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.672180272Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-20T18:55:18.834 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.675002992Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-20T18:55:18.835 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.675362916Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-20T18:55:18.835 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.678110685Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-20T18:55:18.835 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.678703392Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-20T18:55:18.835 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.680966422Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.683856905Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.686626603Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.689322252Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.69202349Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.694780421Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.697433395Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.700196875Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.702914195Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.705459854Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.708208122Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-20T18:55:18.836 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.710882861Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.713761768Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.716563195Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.719257764Z level=info msg="Executing migration" id="create quota table v1" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.72210785Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.724763886Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.727504731Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.72804699Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.731086317Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.733341218Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.734115693Z level=info msg="Executing migration" id="create session table" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.734576833Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.734852719Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.735312043Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.735688434Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.736628312Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-20T18:55:18.837 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.737095865Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.737373692Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.737639669Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.737904603Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.738287604Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.738543166Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.739144633Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.73944685Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.7400331Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.740815477Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.741604678Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.742044611Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.742857788Z level=info msg="Executing migration" id="create alert table v1" 2024-09-20T18:55:18.838 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.743750903Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.744465543Z level=info msg="Executing migration" id="add index alert state" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.744868726Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.745269402Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.745635216Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.746097887Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.746484065Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.747906401Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.748279735Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.748693496Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.749008752Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.749357171Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.749698536Z level=info msg="Executing migration" id="Add column is_default" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.750312489Z level=info msg="Executing migration" id="Add column frequency" 2024-09-20T18:55:18.839 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.750927862Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.751558174Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.752171153Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.752576189Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.752823938Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.753133673Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.753511777Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.753921753Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.754294742Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.754644291Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.755094561Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.756732787Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.757388242Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.757698543Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.758121045Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-20T18:55:18.840 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.758505507Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.759194459Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.75950172Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.759894041Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.760301994Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.760578203Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.761014261Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.761783655Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.762218703Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.762612671Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.763073087Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.763866649Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-20T18:55:18.841 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.764750682Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.765571894Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.766558231Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.767409161Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.768269429Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.769120415Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.769933307Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.771556589Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.772398475Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.773278113Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.774088616Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.775045737Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.775815927Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.776669856Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-20T18:55:18.842 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.777646568Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.778404041Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.77924398Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.780076024Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.780941835Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.781814631Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.782700501Z level=info msg="Executing migration" id="Move region to single row" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.783658961Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.786384039Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.789027244Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.7918234Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.794518231Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.794953511Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-20T18:55:18.843 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.797771615Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.800516482Z level=info msg="Executing migration" id="create test_data table" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.803234003Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.803639669Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.806522289Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.809324786Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.812342752Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.812807975Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.815593472Z level=info msg="Executing migration" id="create team table" 2024-09-20T18:55:18.844 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.818443387Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-20T18:55:19.077 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-09-20T18:55:19.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi076", "caps": []}]: dispatch 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.821282065Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.821756632Z level=info msg="Executing migration" id="create team member table" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.824496878Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.82727458Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.829881441Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.830340323Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.832765694Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.835524021Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.838346731Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.838910591Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.841475322Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-20T18:55:19.079 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.844117818Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.846795601Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.849387395Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.85208875Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.854922288Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.857686969Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.860269983Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.86289381Z level=info msg="Executing migration" id="create tag table" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.865494754Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.866183989Z level=info msg="Executing migration" id="create login attempt table" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.867033664Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.868010503Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.868687683Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.870662684Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-20T18:55:19.080 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.871494296Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.872489879Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.873197059Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.874956691Z level=info msg="Executing migration" id="create user auth table" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.875754399Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.876756494Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.877510547Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.878359067Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.879198466Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.880051992Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.88093693Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.881805487Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-20T18:55:19.081 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.882670612Z level=info msg="Executing migration" id="create server_lock table" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.883696909Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.884425818Z level=info msg="Executing migration" id="create user auth token table" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.885383897Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.886175576Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.886994076Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.887872829Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-20T18:55:19.082 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.888695393Z level=info msg="Executing migration" id="create cache_data table" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.889673126Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.890434105Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.891321635Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.892201825Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.89304632Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.894031724Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.894758791Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.895761791Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.89651579Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.897325903Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.897765966Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.900236946Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.902953496Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.905766164Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-20T18:55:19.083 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.908793624Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.911484862Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.914573079Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.917088843Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.919912875Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.92288949Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.925886167Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.928727707Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.931493047Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.934280124Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.937085011Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.939900967Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.942709888Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.945355819Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.948783566Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.952106922Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-20T18:55:19.084 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.954550608Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.957246077Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.96001203Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.962435145Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.965056879Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.967741906Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.970479785Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.973048478Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.975954914Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.978642703Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.981318988Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.984014156Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.986553259Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.989044892Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.991719187Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-20T18:55:19.085 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.992582579Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.993421989Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.994309453Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.995134056Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.995982862Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.996823454Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.997705118Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.998523712Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:18.999454967Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.000367877Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.001239605Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.003085975Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.004002278Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.004829808Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.005701647Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-20T18:55:19.086 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.00657638Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.007424634Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.008371638Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.009268667Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.010147886Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.011006778Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.011863465Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.012742102Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.013613058Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-20T18:55:19.087 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.014510025Z level=info msg="Executing migration" id="create alert_image table" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.015356664Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.016268715Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.017099856Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.019405457Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.01990571Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.020010046Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.02271839Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.025492022Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.028239545Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.028384758Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.028523516Z level=info msg="alerts found to migrate" alerts=0 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.030873882Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.033681829Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.036341598Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-20T18:55:19.088 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.038976849Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.041656111Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.044432561Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.047126735Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.049793855Z level=info msg="Executing migration" id="create data_keys table" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.052604369Z level=info msg="Executing migration" id="create secrets table" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.055407608Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.059762454Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.060700195Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.063328549Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-20T18:55:19.089 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.067722607Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-20T18:55:19.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-09-20T18:55:19.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi076", "caps": []}]: dispatch 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-09-20T18:55:19.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:19 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi076", "caps": []}]: dispatch 2024-09-20T18:55:19.344 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.072240285Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-20T18:55:19.344 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.072706578Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-20T18:55:19.344 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.073505794Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.074301052Z level=info msg="Executing migration" id="create permission table" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.075090021Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.075857412Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.076671434Z level=info msg="Executing migration" id="create role table" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.077443933Z level=info msg="Executing migration" id="add column display_name" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.07843437Z level=info msg="Executing migration" id="add column group_name" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.07942842Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.080218191Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.08103382Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.081861394Z level=info msg="Executing migration" id="create team role table" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.082264388Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-20T18:55:19.345 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.083084623Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.08387033Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.084824747Z level=info msg="Executing migration" id="create user role table" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.085536496Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.086474886Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.087232464Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.088032219Z level=info msg="Executing migration" id="create builtin role table" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.088494679Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.089422986Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.090157706Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.091163939Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.091631299Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.092441417Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.092887908Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-20T18:55:19.346 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.093331476Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.093718065Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.094166903Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.095160983Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.095577344Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.096057576Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.096359123Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.096749465Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.097215147Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-20T18:55:19.347 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.097596177Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.097889182Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.098259782Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.098656279Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.099106459Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.100081151Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.100388493Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.100805835Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-20T18:55:19.348 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.101248722Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.101680472Z level=info msg="Executing migration" id="add correlation config column" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.102677535Z level=info msg="Executing migration" id="create entity_events table" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.103136381Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.103915081Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.104025699Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.104389621Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.104480165Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.104738236Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.105162016Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.105562654Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.106000455Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.106424561Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-20T18:55:19.349 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.106813072Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.107242074Z level=info msg="Executing migration" id="Drop public config table" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.10761684Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.10804425Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.108462324Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.108887777Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.10930607Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.112333897Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.113357851Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.114407836Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.114718836Z level=info msg="Executing migration" id="add share column" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.115712088Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.116160925Z level=info msg="Executing migration" id="create file table" 2024-09-20T18:55:19.350 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.116588862Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.117031969Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.117488242Z level=info msg="Executing migration" id="create file_meta table" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.118337255Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.119187532Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.12001903Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.120404627Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.12125681Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.121699225Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.122743652Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.125426885Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.125960694Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.128767635Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.131481674Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.134195657Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-20T18:55:19.351 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.13533633Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.138116634Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.140809188Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.150543556Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.151050242Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.153946578Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.157305198Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.160097353Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.160502564Z level=info msg="Executing migration" id="create folder table" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.163215365Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.163688619Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.166097847Z level=info msg="Executing migration" id="Update folder title length" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.168885575Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=migrator t=2024-09-20T18:55:19.169376627Z level=info msg="migrations completed" performed=485 skipped=0 duration=692.522435ms 2024-09-20T18:55:19.352 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=sqlstore t=2024-09-20T18:55:19.16979947Z level=info msg="Created default organization" 2024-09-20T18:55:19.353 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=plugin.loader t=2024-09-20T18:55:19.243552513Z level=info msg="Plugin registered" pluginID=input 2024-09-20T18:55:19.353 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=plugin.loader t=2024-09-20T18:55:19.276025152Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-20T18:55:19.353 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=plugin.loader t=2024-09-20T18:55:19.276056158Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-20T18:55:19.353 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=secrets t=2024-09-20T18:55:19.27610176Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-20T18:55:19.353 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=query_data t=2024-09-20T18:55:19.278507539Z level=info msg="Query Service initialization" 2024-09-20T18:55:19.353 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=live.push_http t=2024-09-20T18:55:19.283661927Z level=info msg="Live Push Gateway initialization" 2024-09-20T18:55:20.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[23648]: Reconfiguring daemon agent.smithi076 on smithi076 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[30031]: Reconfiguring daemon agent.smithi076 on smithi076 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-mon[25770]: Reconfiguring daemon agent.smithi076 on smithi076 2024-09-20T18:55:20.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:20.340 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=infra.usagestats.collector t=2024-09-20T18:55:20.051600472Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-20T18:55:20.340 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=provisioning.datasources t=2024-09-20T18:55:20.107682869Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T18:55:20.340 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=provisioning.datasources t=2024-09-20T18:55:20.140521569Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=provisioning.alerting t=2024-09-20T18:55:20.172561296Z level=info msg="starting to provision alerting" 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=provisioning.alerting t=2024-09-20T18:55:20.172597307Z level=info msg="finished to provision alerting" 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=grafanaStorageLogger t=2024-09-20T18:55:20.172793358Z level=info msg="storage starting" 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=ngalert.state.manager t=2024-09-20T18:55:20.172823877Z level=info msg="Warming state cache for startup" 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=http.server t=2024-09-20T18:55:20.174536422Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=sqlstore.transactions t=2024-09-20T18:55:20.183067668Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=ngalert.state.manager t=2024-09-20T18:55:20.18308665Z level=info msg="State cache has been initialized" states=0 duration=10.261462ms 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=ngalert.multiorg.alertmanager t=2024-09-20T18:55:20.183143416Z level=info msg="starting MultiOrg Alertmanager" 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=ticker t=2024-09-20T18:55:20.183165548Z level=info msg=starting first_tick=2024-09-20T18:55:30Z 2024-09-20T18:55:20.341 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 18:55:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a[47735]: logger=sqlstore.transactions t=2024-09-20T18:55:20.220329126Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T18:55:20.538 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:20 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:55:20] "GET /metrics HTTP/1.1" 200 26195 "" "Prometheus/2.43.0" 2024-09-20T18:55:21.225 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:21 smithi076 ceph-mon[23648]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:21.225 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:21 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:21.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:21 smithi076 ceph-mon[30031]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:21.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:21 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:21.267 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:21 smithi194 ceph-mon[25770]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:21.267 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:21 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[23648]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:23.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[30031]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:23.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:23 smithi194 ceph-mon[25770]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:23.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:23.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:25.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:25 smithi194 ceph-mon[25770]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:25.355 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:25 smithi076 ceph-mon[23648]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:25.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:25 smithi076 ceph-mon[30031]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:25.988 INFO:teuthology.orchestra.run.smithi076.stdout:107374182435 2024-09-20T18:55:25.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T18:55:26.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:26 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:26.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:26 smithi076 ceph-mon[23648]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:26.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:26 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:26.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:26 smithi076 ceph-mon[30031]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:27.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:26 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:26 smithi194 ceph-mon[25770]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:27.588 INFO:teuthology.orchestra.run.smithi076.stdout:231928234000 2024-09-20T18:55:27.589 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T18:55:27.878 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.879 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:27.901 INFO:teuthology.orchestra.run.smithi076.stdout:55834574894 2024-09-20T18:55:27.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T18:55:28.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:28.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:28.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:28.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:28.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:28.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:28 smithi076 ceph-mon[23648]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:28.947 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:28 smithi076 ceph-mon[30031]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:28.985 INFO:teuthology.orchestra.run.smithi076.stdout:34359738417 2024-09-20T18:55:28.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T18:55:29.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:28 smithi194 ceph-mon[25770]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:29.398 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:30.787 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:55:30] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-20T18:55:30.983 INFO:teuthology.orchestra.run.smithi076.stdout:163208757275 2024-09-20T18:55:30.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T18:55:31.050 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:31 smithi076 ceph-mon[30031]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:31.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:31 smithi194 ceph-mon[25770]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:31.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:31 smithi076 ceph-mon[23648]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:32.323 INFO:teuthology.orchestra.run.smithi076.stdout:128849018912 2024-09-20T18:55:32.323 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T18:55:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:197568495638 2024-09-20T18:55:32.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T18:55:33.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:33 smithi076 ceph-mon[23648]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:33.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:33 smithi076 ceph-mon[30031]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:33.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:33 smithi194 ceph-mon[25770]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:35.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:35 smithi076 ceph-mon[23648]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:35.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:35 smithi076 ceph-mon[30031]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:35.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:35 smithi194 ceph-mon[25770]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:36.805 INFO:teuthology.orchestra.run.smithi076.stdout:77309411370 2024-09-20T18:55:36.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T18:55:37.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:37 smithi076 ceph-mon[30031]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:37.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:37 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2818498836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T18:55:37.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:37 smithi076 ceph-mon[23648]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:37.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:37 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2818498836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T18:55:37.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:37 smithi194 ceph-mon[25770]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:37.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:37 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2818498836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T18:55:37.773 INFO:teuthology.orchestra.run.smithi076.stdout:107374182437 2024-09-20T18:55:37.971 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:38.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182437 for osd.3 2024-09-20T18:55:38.365 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:38.638 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:39.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:39 smithi076 ceph-mon[23648]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:39.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:39 smithi076 ceph-mon[30031]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:39.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:39 smithi194 ceph-mon[25770]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:39.517 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:39.700 INFO:teuthology.orchestra.run.smithi076.stdout:231928234003 2024-09-20T18:55:40.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:40 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4113160707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T18:55:40.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:40 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4113160707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T18:55:40.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:40 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4113160707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T18:55:40.787 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:55:40] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-20T18:55:40.841 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234003 for osd.7 2024-09-20T18:55:40.841 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:41.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:41 smithi076 ceph-mon[23648]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:41.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:41 smithi076 ceph-mon[30031]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:41.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:41 smithi194 ceph-mon[25770]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:41.498 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:42.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:42.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:42.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:42.770 INFO:teuthology.orchestra.run.smithi076.stdout:128849018914 2024-09-20T18:55:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:55834574897 2024-09-20T18:55:43.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:43 smithi076 ceph-mon[23648]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:43.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:43 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/88916637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T18:55:43.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:43 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/218400020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T18:55:43.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:43 smithi076 ceph-mon[30031]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:43.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:43 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/88916637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T18:55:43.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:43 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/218400020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T18:55:43.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:43 smithi194 ceph-mon[25770]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:43.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:43 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/88916637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T18:55:43.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:43 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/218400020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T18:55:45.343 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:45 smithi194 ceph-mon[25770]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:45.480 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:45 smithi076 ceph-mon[23648]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:45.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:45 smithi076 ceph-mon[30031]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:45.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574897 for osd.1 2024-09-20T18:55:45.505 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:47.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:47 smithi194 ceph-mon[25770]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:47.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:47 smithi076 ceph-mon[23648]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:47.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:47 smithi076 ceph-mon[30031]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:48.244 INFO:teuthology.orchestra.run.smithi076.stdout:163208757278 2024-09-20T18:55:48.645 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018912 got 128849018914 for osd.4 2024-09-20T18:55:48.646 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:48.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[30031]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:48.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4208196323' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T18:55:48.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[23648]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:48.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:48 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4208196323' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T18:55:48.842 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:48 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.843 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:48 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.843 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:48 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:55:48.843 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:48 smithi194 ceph-mon[25770]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:48.843 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:48 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4208196323' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T18:55:48.989 INFO:teuthology.orchestra.run.smithi076.stdout:34359738423 2024-09-20T18:55:49.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738423 for osd.0 2024-09-20T18:55:49.753 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:49.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:49 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/279711415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T18:55:49.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:49 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/279711415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T18:55:49.842 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:49 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/279711415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T18:55:50.098 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757275 got 163208757278 for osd.5 2024-09-20T18:55:50.098 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:50.281 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:50.450 INFO:teuthology.orchestra.run.smithi076.stdout:197568495641 2024-09-20T18:55:50.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:50 smithi076 ceph-mon[23648]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:50.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:50 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1460891922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T18:55:50.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:50 smithi076 ceph-mon[30031]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:50.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:50 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1460891922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T18:55:50.788 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:55:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:55:50] "GET /metrics HTTP/1.1" 200 26202 "" "Prometheus/2.43.0" 2024-09-20T18:55:50.842 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:50 smithi194 ceph-mon[25770]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:50.842 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:50 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1460891922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T18:55:51.389 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495638 got 197568495641 for osd.6 2024-09-20T18:55:51.389 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:53.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:53 smithi076 ceph-mon[23648]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:53.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:53 smithi076 ceph-mon[30031]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:53.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:53 smithi194 ceph-mon[25770]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:55.245 INFO:teuthology.orchestra.run.smithi076.stdout:77309411375 2024-09-20T18:55:55.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:55 smithi076 ceph-mon[23648]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:55.245 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:55 smithi076 ceph-mon[30031]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:55.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:55 smithi194 ceph-mon[25770]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:56.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411370 got 77309411375 for osd.2 2024-09-20T18:55:56.014 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:56.014 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T18:55:56.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T18:55:56.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:56 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/776203522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T18:55:56.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:56 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/776203522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T18:55:56.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:56 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/776203522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T18:55:56.664 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:55:57.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:57 smithi076 ceph-mon[23648]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:57.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:57.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:57 smithi076 ceph-mon[30031]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:57.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:57.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:57 smithi194 ceph-mon[25770]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:57.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:55:59.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:55:59 smithi194 ceph-mon[25770]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:59.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:55:59 smithi076 ceph-mon[23648]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:55:59.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:55:59 smithi076 ceph-mon[30031]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:00.249 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:00.249 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-20T18:56:00.537 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:56:00] "GET /metrics HTTP/1.1" 200 26198 "" "Prometheus/2.43.0" 2024-09-20T18:56:01.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:01 smithi076 ceph-mon[30031]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:01.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:01 smithi076 ceph-mon[30031]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:56:01.204 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:01 smithi076 ceph-mon[23648]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:01.204 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:01 smithi076 ceph-mon[23648]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:56:01.332 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:01 smithi194 ceph-mon[25770]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:01.333 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:01 smithi194 ceph-mon[25770]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:56:01.686 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-20T18:55:59.987714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":222460,"kb_used_data":6304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749738756,"statfs":{"total":767960285184,"available":767732486144,"internally_reserved":0,"allocated":6455296,"data_stored":3678675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002039"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":255,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-20T18:54:44.038514+0000","last_change":"2024-09-20T18:54:15.649222+0000","last_active":"2024-09-20T18:54:44.038514+0000","last_peered":"2024-09-20T18:54:44.038514+0000","last_clean":"2024-09-20T18:54:44.038514+0000","last_became_active":"2024-09-20T18:54:15.648913+0000","last_became_peered":"2024-09-20T18:54:15.648913+0000","last_unstale":"2024-09-20T18:54:44.038514+0000","last_undegraded":"2024-09-20T18:54:44.038514+0000","last_fullsized":"2024-09-20T18:54:44.038514+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T18:52:14.293649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T18:52:14.293649+0000","last_clean_scrub_stamp":"2024-09-20T18:52:14.293649+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T21:48:13.511254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234008,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":828594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.505}]},{"osd":1,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.52200000000000002}]},{"osd":5,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39800000000000002}]},{"osd":6,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.436},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.47299999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":828925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.59099999999999997,"15min":0.52600000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":9.1170000000000009,"5min":9.1170000000000009,"15min":9.1170000000000009},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.42199999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.61399999999999999,"15min":0.55200000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.57399999999999995}]},{"osd":3,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41499999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.438}]},{"osd":4,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.42999999999999999,"15min":0.41999999999999998},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52000000000000002}]},{"osd":5,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.44400000000000001,"15min":0.434},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.505},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.437,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.626}]},{"osd":7,"last update":"Fri Sep 20 18:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67000000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717564,"statfs":{"total":95995035648,"available":95966785536,"internally_reserved":0,"allocated":581632,"data_stored":238276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":0.65400000000000003,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.722999999999999,"5min":16.722999999999999,"15min":16.722999999999999},"last":0.622},{"interface":"front","average":{"1min":1.302,"5min":0.65400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":16.710000000000001,"5min":16.710000000000001,"15min":16.710000000000001},"last":0.54600000000000004}]},{"osd":1,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.36599999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40600000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.503,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.501,"15min":0.497},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.56000000000000005}]},{"osd":4,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.39600000000000002,"15min":0.39200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.41399999999999998}]},{"osd":6,"last update":"Fri Sep 20 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.437},"min":{"1min":0.309,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46100000000000002}]},{"osd":7,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.438}]}]},{"osd":4,"up_from":30,"seq":128849018918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717564,"statfs":{"total":95995035648,"available":95966785536,"internally_reserved":0,"allocated":581632,"data_stored":238284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.94699999999999995,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79300000000000004,"5min":11.874000000000001,"15min":11.874000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.95499999999999996,"15min":1.026},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":11.840999999999999,"15min":11.840999999999999},"last":0.59999999999999998}]},{"osd":2,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.53600000000000003},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.73499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.95799999999999996,"15min":1.026},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85699999999999998,"5min":11.824,"15min":11.824},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.94799999999999995,"15min":1.012},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":11.863,"15min":11.863},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70799999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.51700000000000002}]},{"osd":5,"last update":"Fri Sep 20 18:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67900000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52600000000000002}]},{"osd":6,"last update":"Fri Sep 20 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.66700000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.68799999999999994}]},{"osd":7,"last update":"Fri Sep 20 18:55:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66200000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.47399999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.32700000000000001}]},{"osd":1,"last update":"Fri Sep 20 18:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.37,"15min":0.37},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.40300000000000002,"15min":0.40500000000000003},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41599999999999998}]},{"osd":2,"last update":"Fri Sep 20 18:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.437,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.63,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.372}]},{"osd":4,"last update":"Fri Sep 20 18:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.501,"15min":0.48699999999999999},"min":{"1min":0.45700000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.40899999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Sep 20 18:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73099999999999998,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.626},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.875,"15min":0.93200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.746,"5min":8.9600000000000009,"15min":8.9600000000000009},"last":0.69199999999999995}]},{"osd":6,"last update":"Fri Sep 20 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72099999999999997,"5min":0.752,"15min":0.752},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74199999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67100000000000004}]},{"osd":7,"last update":"Fri Sep 20 18:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.74199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33700000000000002,"15min":0.34000000000000002},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.46100000000000002,"5min":0.499,"15min":0.499},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34899999999999998,"15min":0.34499999999999997},"min":{"1min":0.25,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.316}]},{"osd":1,"last update":"Fri Sep 20 18:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.41699999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.378}]},{"osd":3,"last update":"Fri Sep 20 18:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39400000000000002},"min":{"1min":0.29299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.56499999999999995,"5min":0.621,"15min":0.621},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.376,"15min":0.38},"min":{"1min":0.29699999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.5,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.34899999999999998}]},{"osd":4,"last update":"Fri Sep 20 18:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.84999999999999998}]},{"osd":5,"last update":"Fri Sep 20 18:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60099999999999998,"15min":0.61199999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69899999999999995,"5min":0.878,"15min":0.878},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.73899999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.78700000000000003}]},{"osd":6,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77600000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Sep 20 18:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35299999999999998,"15min":0.34000000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35299999999999998,"15min":0.34499999999999997},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51600000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.33300000000000002}]},{"osd":2,"last update":"Fri Sep 20 18:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.374},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54400000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.375,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53200000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.35099999999999998}]},{"osd":3,"last update":"Fri Sep 20 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60699999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.41599999999999998,"15min":0.42699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56299999999999994,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.376}]},{"osd":4,"last update":"Fri Sep 20 18:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":0.87,"15min":0.87},"last":0.59299999999999997}]},{"osd":5,"last update":"Fri Sep 20 18:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.67400000000000004,"15min":0.69799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76100000000000001,"5min":3.8319999999999999,"15min":3.8319999999999999},"last":0.78100000000000003}]},{"osd":6,"last update":"Fri Sep 20 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77900000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.748},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82299999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Fri Sep 20 18:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":828925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 18:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.31,"15min":0.29199999999999998},"min":{"1min":0.245,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.29999999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54300000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Sep 20 18:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36199999999999999,"15min":0.35899999999999999},"min":{"1min":0.253,"5min":0.222,"15min":0.222},"max":{"1min":0.54100000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.379},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Sep 20 18:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44500000000000001,"15min":0.45700000000000002},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41799999999999998,"15min":0.43099999999999999},"min":{"1min":0.27500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.78100000000000003,"15min":0.63600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":18.815000000000001,"5min":18.815000000000001,"15min":18.815000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Sep 20 18:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.622,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":3.536,"5min":3.536,"15min":3.536},"last":0.629},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Sep 20 18:54:52 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.777000000000001,"5min":18.777000000000001,"15min":18.777000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.76300000000000001}]},{"osd":7,"last update":"Fri Sep 20 18:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T18:56:01.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T18:56:02.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:02 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.443 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:02 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.443 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:02 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.443 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:02 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:02.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:02 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:03.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:03 smithi076 ceph-mon[23648]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:03.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:03 smithi076 ceph-mon[30031]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:03.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:03 smithi194 ceph-mon[25770]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:05.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:05 smithi076 ceph-mon[23648]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:05.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:05 smithi076 ceph-mon[30031]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:05.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:05 smithi194 ceph-mon[25770]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:07.351 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:07.352 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-20T18:56:07.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:07 smithi076 ceph-mon[23648]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:07.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:07 smithi076 ceph-mon[30031]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:07.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:07 smithi194 ceph-mon[25770]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:08.075 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-20T18:56:05.988890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":222460,"kb_used_data":6304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749738756,"statfs":{"total":767960285184,"available":767732486144,"internally_reserved":0,"allocated":6455296,"data_stored":3678675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002293"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":255,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-20T18:54:44.038514+0000","last_change":"2024-09-20T18:54:15.649222+0000","last_active":"2024-09-20T18:54:44.038514+0000","last_peered":"2024-09-20T18:54:44.038514+0000","last_clean":"2024-09-20T18:54:44.038514+0000","last_became_active":"2024-09-20T18:54:15.648913+0000","last_became_peered":"2024-09-20T18:54:15.648913+0000","last_unstale":"2024-09-20T18:54:44.038514+0000","last_undegraded":"2024-09-20T18:54:44.038514+0000","last_fullsized":"2024-09-20T18:54:44.038514+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T18:52:14.293649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T18:52:14.293649+0000","last_clean_scrub_stamp":"2024-09-20T18:52:14.293649+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T21:48:13.511254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":828594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42599999999999999}]},{"osd":3,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.39100000000000001}]},{"osd":4,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54400000000000004}]},{"osd":5,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.44500000000000001}]},{"osd":6,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.46899999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495645,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":828925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.59099999999999997,"15min":0.52600000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":9.1170000000000009,"5min":9.1170000000000009,"15min":9.1170000000000009},"last":0.54900000000000004}]},{"osd":1,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.42199999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.61399999999999999,"15min":0.55200000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.58699999999999997}]},{"osd":3,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41499999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64300000000000002}]},{"osd":4,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.42999999999999999,"15min":0.41999999999999998},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.37}]},{"osd":5,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.44400000000000001,"15min":0.434},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.437,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Sep 20 18:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.498},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.53100000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717564,"statfs":{"total":95995035648,"available":95966785536,"internally_reserved":0,"allocated":581632,"data_stored":238276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":0.65400000000000003,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.722999999999999,"5min":16.722999999999999,"15min":16.722999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.302,"5min":0.65400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":16.710000000000001,"5min":16.710000000000001,"15min":16.710000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51900000000000002,"15min":0.51800000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.36599999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.503}]},{"osd":2,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40600000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47699999999999998}]},{"osd":3,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.503,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.501,"15min":0.497},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Sep 20 18:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.39600000000000002,"15min":0.39200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38200000000000001}]},{"osd":6,"last update":"Fri Sep 20 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.437},"min":{"1min":0.309,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41199999999999998}]},{"osd":7,"last update":"Fri Sep 20 18:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45000000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717564,"statfs":{"total":95995035648,"available":95966785536,"internally_reserved":0,"allocated":581632,"data_stored":238284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.94699999999999995,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79300000000000004,"5min":11.874000000000001,"15min":11.874000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.443}]},{"osd":1,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.95499999999999996,"15min":1.026},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":11.840999999999999,"15min":11.840999999999999},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.53600000000000003},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.73499999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.95799999999999996,"15min":1.026},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85699999999999998,"5min":11.824,"15min":11.824},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.94799999999999995,"15min":1.012},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81699999999999995,"5min":11.863,"15min":11.863},"last":0.378},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70799999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.5}]},{"osd":5,"last update":"Fri Sep 20 18:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67900000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.33600000000000002}]},{"osd":6,"last update":"Fri Sep 20 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.66700000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.48699999999999999}]},{"osd":7,"last update":"Fri Sep 20 18:55:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.436},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.47399999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.29799999999999999}]},{"osd":1,"last update":"Fri Sep 20 18:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.37,"15min":0.37},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.40300000000000002,"15min":0.40500000000000003},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Sep 20 18:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.437,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.63,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.33900000000000002}]},{"osd":4,"last update":"Fri Sep 20 18:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.501,"15min":0.48699999999999999},"min":{"1min":0.45700000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.436},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.40899999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41299999999999998}]},{"osd":5,"last update":"Fri Sep 20 18:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73099999999999998,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.875,"15min":0.93200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.746,"5min":8.9600000000000009,"15min":8.9600000000000009},"last":0.375}]},{"osd":6,"last update":"Fri Sep 20 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72099999999999997,"5min":0.752,"15min":0.752},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74199999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42099999999999999}]},{"osd":7,"last update":"Fri Sep 20 18:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.36699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33700000000000002,"15min":0.34000000000000002},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.46100000000000002,"5min":0.499,"15min":0.499},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34899999999999998,"15min":0.34499999999999997},"min":{"1min":0.25,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Fri Sep 20 18:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.41699999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Sep 20 18:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39400000000000002},"min":{"1min":0.29299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.56499999999999995,"5min":0.621,"15min":0.621},"last":0.374},{"interface":"front","average":{"1min":0.371,"5min":0.376,"15min":0.38},"min":{"1min":0.29699999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.5,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Sep 20 18:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68300000000000005}]},{"osd":5,"last update":"Fri Sep 20 18:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60099999999999998,"15min":0.61199999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69899999999999995,"5min":0.878,"15min":0.878},"last":0.628},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.73899999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Sep 20 18:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77600000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Sep 20 18:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 18:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35299999999999998,"15min":0.34000000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35299999999999998,"15min":0.34499999999999997},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51600000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Sep 20 18:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.374},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54400000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.375},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.375,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53200000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.42599999999999999}]},{"osd":3,"last update":"Fri Sep 20 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60699999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.41599999999999998,"15min":0.42699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56299999999999994,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Fri Sep 20 18:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":0.87,"15min":0.87},"last":0.751}]},{"osd":5,"last update":"Fri Sep 20 18:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.67400000000000004,"15min":0.69799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76100000000000001,"5min":3.8319999999999999,"15min":3.8319999999999999},"last":0.72799999999999998}]},{"osd":6,"last update":"Fri Sep 20 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77900000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82299999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60999999999999999}]},{"osd":7,"last update":"Fri Sep 20 18:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.79700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":828925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 18:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.32800000000000001,"15min":0.29799999999999999},"min":{"1min":0.253,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.30499999999999999},"min":{"1min":0.26400000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Sep 20 18:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36199999999999999,"15min":0.35899999999999999},"min":{"1min":0.253,"5min":0.222,"15min":0.222},"max":{"1min":0.54100000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.379},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.223}]},{"osd":3,"last update":"Fri Sep 20 18:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44500000000000001,"15min":0.45700000000000002},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41799999999999998,"15min":0.43099999999999999},"min":{"1min":0.27500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.28100000000000003}]},{"osd":4,"last update":"Fri Sep 20 18:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.78100000000000003,"15min":0.63600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":18.815000000000001,"5min":18.815000000000001,"15min":18.815000000000001},"last":0.376},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.443}]},{"osd":5,"last update":"Fri Sep 20 18:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.622,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":3.536,"5min":3.536,"15min":3.536},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.38300000000000001}]},{"osd":6,"last update":"Fri Sep 20 18:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.3100000000000001,"15min":1.4259999999999999},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91900000000000004,"5min":18.777000000000001,"15min":18.777000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58099999999999996,"15min":0.57799999999999996},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.47399999999999998}]},{"osd":7,"last update":"Fri Sep 20 18:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T18:56:08.077 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T18:56:08.077 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T18:56:08.077 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T18:56:08.078 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:08.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[23648]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[23648]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[30031]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:08.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:08 smithi076 ceph-mon[30031]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:09.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:08 smithi194 ceph-mon[25770]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T18:56:09.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:08 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:08 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:08 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:08 smithi194 ceph-mon[25770]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: Detected new or changed devices on smithi076 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T18:56:09.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[30031]: Reconfiguring daemon alertmanager.a on smithi076 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: Detected new or changed devices on smithi076 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T18:56:09.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:09 smithi076 ceph-mon[23648]: Reconfiguring daemon alertmanager.a on smithi076 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: Detected new or changed devices on smithi076 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T18:56:10.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:09 smithi194 ceph-mon[25770]: Reconfiguring daemon alertmanager.a on smithi076 2024-09-20T18:56:10.690 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:56:10] "GET /metrics HTTP/1.1" 200 26198 "" "Prometheus/2.43.0" 2024-09-20T18:56:11.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:10 smithi076 ceph-mon[23648]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T18:56:11.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:10 smithi076 ceph-mon[30031]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T18:56:11.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:10 smithi194 ceph-mon[25770]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T18:56:11.887 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:11.887 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:12.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:11 smithi076 ceph-mon[23648]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:12.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:11 smithi076 ceph-mon[30031]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:12.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:11 smithi194 ceph-mon[25770]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:12.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:12 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2197640433' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:12.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:12 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:12.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:12 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:12.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:12 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2197640433' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:12.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:12 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:12.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:12 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:13.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:12 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2197640433' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:13.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:12 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:13.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:12 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:13.529 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:13 smithi076 systemd[1]: Stopping Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:56:13.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:13 smithi076 ceph-mon[23648]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:13.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:13 smithi076 ceph-mon[30031]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:14.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:13 smithi194 ceph-mon[25770]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:14.146 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[50033]: ts=2024-09-20T18:56:14.112Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T18:56:14.470 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:14 smithi076 podman[59794]: 2024-09-20 18:56:14.133092167 +0000 UTC m=+0.129107044 container died 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:14.787 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:14 smithi076 podman[59794]: 2024-09-20 18:56:14.654876575 +0000 UTC m=+0.650891445 container cleanup 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:14.787 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:14 smithi076 bash[59794]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T18:56:15.241 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:14 smithi076 podman[59806]: 2024-09-20 18:56:14.865095378 +0000 UTC m=+0.728103745 container remove 3858fd48fb80bc6fae9b7793ec98110de648988cca7be24d8e67b93a8fba0dec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:15.241 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:14 smithi076 podman[59806]: 2024-09-20 18:56:14.949038521 +0000 UTC m=+0.812046842 volume remove 0f15fafec5a824fac564f46104d2823e79e57df01e181f1caf11119f5293c684 2024-09-20T18:56:16.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:15 smithi076 ceph-mon[30031]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:16.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:15 smithi076 ceph-mon[23648]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:16.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:15 smithi194 ceph-mon[25770]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:16.522 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T18:56:16.522 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 systemd[1]: Stopped Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:56:16.522 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 systemd[1]: Starting Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:56:16.786 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 podman[59932]: 2024-09-20 18:56:16.409805498 +0000 UTC m=+0.015378315 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-20T18:56:16.786 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 podman[59932]: 2024-09-20 18:56:16.521537948 +0000 UTC m=+0.127110800 volume create 5bd3e566ee02a8d0c7f40e7a62e9218a055b00d8e1dcbc8e6d2b4e50264caf5f 2024-09-20T18:56:16.786 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 podman[59932]: 2024-09-20 18:56:16.621856357 +0000 UTC m=+0.227429163 container create 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:16.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 podman[59932]: 2024-09-20 18:56:16.907412938 +0000 UTC m=+0.512985747 container init 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 podman[59932]: 2024-09-20 18:56:16.910169236 +0000 UTC m=+0.515742043 container start 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.922Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.922Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.923Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.76 port=9094 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.924Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.948Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.948Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.951Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:16 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:16.951Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T18:56:17.212 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:17 smithi076 bash[59932]: 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 2024-09-20T18:56:17.213 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:17 smithi076 systemd[1]: Started Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:56:17.564 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:18.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:18 smithi194 ceph-mon[25770]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:18.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:18 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:18.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:18 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:18.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:18 smithi194 ceph-mon[25770]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T18:56:18.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:18 smithi194 ceph-mon[25770]: Reconfiguring daemon prometheus.a on smithi194 2024-09-20T18:56:18.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[23648]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[23648]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[23648]: Reconfiguring daemon prometheus.a on smithi194 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[30031]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[30031]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T18:56:18.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-mon[30031]: Reconfiguring daemon prometheus.a on smithi194 2024-09-20T18:56:19.036 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:18 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:18.925Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00048622s 2024-09-20T18:56:20.248 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-mon[25770]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:20.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:20 smithi076 ceph-mon[23648]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:20.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:20 smithi076 ceph-mon[30031]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:20.327 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:20.327 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:20.591 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 systemd[1]: Stopping Ceph prometheus.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:56:20.786 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:20 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:56:20] "GET /metrics HTTP/1.1" 200 26271 "" "Prometheus/2.43.0" 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T18:56:20.953 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T18:56:20.954 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.664Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T18:56:20.954 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.665Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T18:56:20.954 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.665Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T18:56:20.954 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[44772]: ts=2024-09-20T18:56:20.665Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T18:56:20.954 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:20 smithi194 podman[49466]: 2024-09-20 18:56:20.674824787 +0000 UTC m=+0.142045064 container died 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:21.213 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:21 smithi194 podman[49466]: 2024-09-20 18:56:21.047357929 +0000 UTC m=+0.514578205 container cleanup 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:21.213 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:21 smithi194 bash[49466]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T18:56:21.213 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:21 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/236986011' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:21.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:21 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/236986011' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:21.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:21 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/236986011' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:22.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-mon[25770]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:22.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:22 smithi076 ceph-mon[30031]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:22.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:22 smithi076 ceph-mon[23648]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:22.618 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 podman[49478]: 2024-09-20 18:56:22.346547864 +0000 UTC m=+1.673834519 container remove 37409337691b759a8c1225a11c749454d00b17ea435348cd041ef221d9cf0de9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 bash[49651]: Error: no container with name or ID "ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a" found: no such container 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 systemd[1]: Stopped Ceph prometheus.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.158s CPU time. 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 systemd[1]: Starting Ceph prometheus.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 podman[49732]: 2024-09-20 18:56:22.494017052 +0000 UTC m=+0.024141381 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T18:56:22.619 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 podman[49732]: 2024-09-20 18:56:22.605818149 +0000 UTC m=+0.135942476 container create a74adbec870e5c87d60a2d886b8e5246f16e0db6a3311028cf810213dd9f42b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 podman[49732]: 2024-09-20 18:56:22.857551781 +0000 UTC m=+0.387676108 container init a74adbec870e5c87d60a2d886b8e5246f16e0db6a3311028cf810213dd9f42b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 podman[49732]: 2024-09-20 18:56:22.860348763 +0000 UTC m=+0.390473093 container start a74adbec870e5c87d60a2d886b8e5246f16e0db6a3311028cf810213dd9f42b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.884Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.884Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.884Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi194 (none))" 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.884Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T18:56:23.142 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.884Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.886Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.886Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.887Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.887Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.890Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.890Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.702µs 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.890Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.891Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.891Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.891Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.164µs wal_replay_duration=1.160036ms wbl_replay_duration=105ns total_replay_duration=1.196958ms 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.892Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.892Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.892Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T18:56:23.143 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.912Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.787604ms db_storage=814ns remote_storage=1.197µs web_handler=355ns query_engine=553ns scrape=386.984µs scrape_sd=119.053µs notify=23.037µs notify_sd=15.998µs rules=18.885611ms tracing=6.638µs 2024-09-20T18:56:23.144 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.912Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T18:56:23.144 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T18:56:22.912Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T18:56:23.144 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 bash[49732]: a74adbec870e5c87d60a2d886b8e5246f16e0db6a3311028cf810213dd9f42b1 2024-09-20T18:56:23.144 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 18:56:22 smithi194 systemd[1]: Started Ceph prometheus.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T18:56:23.286 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STOPPING 2024-09-20T18:56:23.286 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T18:56:23.286 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STOPPED 2024-09-20T18:56:23.286 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STARTING 2024-09-20T18:56:23.656 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Serving on http://:::9283 2024-09-20T18:56:23.657 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STARTED 2024-09-20T18:56:23.657 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STOPPING 2024-09-20T18:56:23.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:23.991 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T18:56:23.991 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STOPPED 2024-09-20T18:56:23.991 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STARTING 2024-09-20T18:56:23.991 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Serving on http://:::9283 2024-09-20T18:56:23.991 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STARTED 2024-09-20T18:56:23.991 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:23] ENGINE Bus STOPPING 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T18:56:24.092 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.093 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:23 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:56:24.168 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:24.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:24.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T18:56:24.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T18:56:24.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:24.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:23 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:56:24.544 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:24 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T18:56:24.544 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:24 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:24] ENGINE Bus STOPPED 2024-09-20T18:56:24.545 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:24 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:24] ENGINE Bus STARTING 2024-09-20T18:56:24.545 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:24 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:24] ENGINE Serving on http://:::9283 2024-09-20T18:56:24.545 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:24 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:18:56:24] ENGINE Bus STARTED 2024-09-20T18:56:26.151 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:26.152 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:26.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:25 smithi076 ceph-mon[23648]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:26.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:25 smithi076 ceph-mon[30031]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:26.311 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:25 smithi194 ceph-mon[25770]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:27.006 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 18:56:26 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T18:56:26.926Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001632333s 2024-09-20T18:56:27.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:26 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1494573747' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:27.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:26 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1494573747' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:27.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:26 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1494573747' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:28.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[23648]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:28.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:28.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[30031]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:28.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:28.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:27 smithi194 ceph-mon[25770]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:28.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:28.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:28.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:56:29.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:56:29.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:29.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:29.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:29 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:29.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:56:29.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:56:29.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:29.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:29.592 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:29 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:30.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:30.744 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:30.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:30 smithi076 ceph-mon[23648]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:30 smithi076 ceph-mon[30031]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:30.786 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:56:30] "GET /metrics HTTP/1.1" 200 26269 "" "Prometheus/2.43.0" 2024-09-20T18:56:30.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:30 smithi194 ceph-mon[25770]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:32.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:32 smithi076 ceph-mon[23648]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:32.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:32 smithi076 ceph-mon[30031]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:32.646 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:32.646 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:32.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:32 smithi194 ceph-mon[25770]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:33.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:33 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3442913242' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:33.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:33 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3442913242' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:33.743 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:33 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3442913242' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:34.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:34 smithi076 ceph-mon[23648]: pgmap v61: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:34.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:34 smithi076 ceph-mon[30031]: pgmap v61: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:34.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:34 smithi194 ceph-mon[25770]: pgmap v61: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:36.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:36.692 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:36 smithi076 ceph-mon[23648]: pgmap v62: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:36.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:36 smithi076 ceph-mon[30031]: pgmap v62: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:36.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:36 smithi194 ceph-mon[25770]: pgmap v62: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:37.929 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:38.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:38 smithi076 ceph-mon[23648]: pgmap v63: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:38.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:38 smithi076 ceph-mon[30031]: pgmap v63: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:38.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:38 smithi194 ceph-mon[25770]: pgmap v63: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:40.479 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:40.479 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:40.480 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:40 smithi076 ceph-mon[23648]: pgmap v64: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:40.480 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:56:40] "GET /metrics HTTP/1.1" 200 26269 "" "Prometheus/2.43.0" 2024-09-20T18:56:40.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:40 smithi076 ceph-mon[30031]: pgmap v64: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:40.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:40 smithi194 ceph-mon[25770]: pgmap v64: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:41.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:41 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2151136558' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:41.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:41 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2151136558' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:41.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:41 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2151136558' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:42.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:42.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:42 smithi194 ceph-mon[25770]: pgmap v65: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:42.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:42.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:42 smithi076 ceph-mon[30031]: pgmap v65: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:42.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:42.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:42 smithi076 ceph-mon[23648]: pgmap v65: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:44.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:44.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:44 smithi076 ceph-mon[23648]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:44.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:44 smithi076 ceph-mon[30031]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:44.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:44 smithi194 ceph-mon[25770]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:46.256 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:46.257 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:46.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:46 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1982655148' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:46.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:46 smithi194 ceph-mon[25770]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:46.859 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.860 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.860 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.860 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1982655148' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:46.860 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[23648]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:46.861 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.861 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.861 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:56:46.861 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1982655148' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:46.861 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:46 smithi076 ceph-mon[30031]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:48.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:48 smithi076 ceph-mon[23648]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:48.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:48 smithi076 ceph-mon[30031]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:48.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:48 smithi194 ceph-mon[25770]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:49.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:50.093 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:50.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:50 smithi076 ceph-mon[23648]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:50.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:56:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:56:50] "GET /metrics HTTP/1.1" 200 26267 "" "Prometheus/2.43.0" 2024-09-20T18:56:50.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:50 smithi076 ceph-mon[30031]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:50.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:50 smithi194 ceph-mon[25770]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:56:51.917 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:56:52.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:51 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/922443488' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:52.221 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:51 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/922443488' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:52.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:51 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/922443488' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:56:53.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:52 smithi076 ceph-mon[23648]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:53.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:52 smithi076 ceph-mon[30031]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:53.341 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:52 smithi194 ceph-mon[25770]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:54.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:54 smithi076 ceph-mon[23648]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:54.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:54 smithi076 ceph-mon[30031]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:54.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:54 smithi194 ceph-mon[25770]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:55.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:56:56.534 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:56:56.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:56 smithi076 ceph-mon[23648]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:56.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:56 smithi076 ceph-mon[30031]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:56.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:56 smithi194 ceph-mon[25770]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:57.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:57.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:57.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:56:58.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:56:58 smithi076 ceph-mon[23648]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:58.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:56:58 smithi076 ceph-mon[30031]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:56:58.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:56:58 smithi194 ceph-mon[25770]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:00.102 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:00.102 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:00.437 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:00 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3493393368' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:00.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:00 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3493393368' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:00.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:00 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3493393368' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:00.746 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:57:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:57:00] "GET /metrics HTTP/1.1" 200 26280 "" "Prometheus/2.43.0" 2024-09-20T18:57:01.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:01 smithi076 ceph-mon[23648]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:01.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:01 smithi076 ceph-mon[30031]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:01.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:01 smithi194 ceph-mon[25770]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:02.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:02 smithi194 ceph-mon[25770]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:02.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:02 smithi076 ceph-mon[23648]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:02.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:02 smithi076 ceph-mon[30031]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:03.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:03.984 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:04.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:04 smithi194 ceph-mon[25770]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:04.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:04 smithi076 ceph-mon[23648]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:04.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:04 smithi076 ceph-mon[30031]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:05.890 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:05.890 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:07.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1238382850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:07.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[23648]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:07.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1238382850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:07.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:06 smithi076 ceph-mon[30031]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:07.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:06 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:06 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:06 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:07.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:06 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1238382850' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:07.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:06 smithi194 ceph-mon[25770]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:08.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:08 smithi076 ceph-mon[23648]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:08.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:08 smithi076 ceph-mon[30031]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:08.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:08 smithi194 ceph-mon[25770]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:09.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:09.836 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:10.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:10 smithi076 ceph-mon[23648]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:10.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:10 smithi076 ceph-mon[30031]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:10.786 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:57:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:57:10] "GET /metrics HTTP/1.1" 200 26280 "" "Prometheus/2.43.0" 2024-09-20T18:57:10.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:10 smithi194 ceph-mon[25770]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:11.676 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:11.676 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:12.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:11 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2410231805' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:12.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:11 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2410231805' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:12.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:11 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2410231805' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:13.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:12 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:13.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:12 smithi076 ceph-mon[30031]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:13.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:12 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:13.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:12 smithi076 ceph-mon[23648]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:13.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:12 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:13.091 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:12 smithi194 ceph-mon[25770]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:14.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:14 smithi076 ceph-mon[23648]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:14.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:14 smithi076 ceph-mon[30031]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:14.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:14 smithi194 ceph-mon[25770]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:15.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:16.618 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:16 smithi076 ceph-mon[23648]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:16.618 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:16 smithi076 ceph-mon[30031]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:16.765 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:16.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:16 smithi194 ceph-mon[25770]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:18.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:18 smithi076 ceph-mon[23648]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:18.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:18 smithi076 ceph-mon[30031]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:18.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:18 smithi194 ceph-mon[25770]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:19.397 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:19.397 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:19.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:19 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3771626585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:19.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:19 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3771626585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:19.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:19 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3771626585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:20.785 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:57:20 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:57:20] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T18:57:21.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:21 smithi194 ceph-mon[25770]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:21.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:21 smithi076 ceph-mon[23648]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:21.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:21 smithi076 ceph-mon[30031]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:23.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:23.243 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:23.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:23 smithi076 ceph-mon[23648]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:23.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:23 smithi076 ceph-mon[30031]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:23.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:23 smithi194 ceph-mon[25770]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:25.155 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:25.156 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:25.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:25 smithi194 ceph-mon[25770]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:25.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:25 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/451262531' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:25.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:25 smithi076 ceph-mon[23648]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:25.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:25 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/451262531' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:25.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:25 smithi076 ceph-mon[30031]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:25.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:25 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/451262531' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[23648]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:26 smithi076 ceph-mon[30031]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:27.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:26 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:26 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:26 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:27.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:26 smithi194 ceph-mon[25770]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:28.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:28.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:28.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:28.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:29.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:28 smithi194 ceph-mon[25770]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:29.090 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:28 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:57:29.103 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:28 smithi076 ceph-mon[30031]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:29.104 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:28 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:57:29.104 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:28 smithi076 ceph-mon[23648]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:29.104 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:28 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:57:29.398 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:30.627 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:30 smithi076 ceph-mon[23648]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:30.628 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:30 smithi076 ceph-mon[30031]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:30.628 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:57:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:57:30] "GET /metrics HTTP/1.1" 200 26280 "" "Prometheus/2.43.0" 2024-09-20T18:57:30.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:30 smithi194 ceph-mon[25770]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:31.150 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:31.150 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:31.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:31 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2212062810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:31.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:31 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2212062810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:31.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:31 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2212062810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:32.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:32 smithi076 ceph-mon[23648]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:32.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:32 smithi076 ceph-mon[30031]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:32.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:32 smithi194 ceph-mon[25770]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:57:34.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:57:34.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:34 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:34 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:34 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:34 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:57:34.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:34 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:57:34.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:34 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:34.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:35.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:35 smithi194 ceph-mon[25770]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:35.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:35 smithi194 ceph-mon[25770]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:35.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:35 smithi076 ceph-mon[23648]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:35.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:35 smithi076 ceph-mon[23648]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:35.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:35 smithi076 ceph-mon[30031]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:35.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:35 smithi076 ceph-mon[30031]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:37.424 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:37 smithi076 ceph-mon[23648]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:37.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:37 smithi076 ceph-mon[30031]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:37.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:37 smithi194 ceph-mon[25770]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:38.182 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:38.182 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:39.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:39 smithi076 ceph-mon[23648]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:39.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:39 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1800752323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:39.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:39 smithi076 ceph-mon[30031]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:39.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:39 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1800752323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:39.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:39 smithi194 ceph-mon[25770]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:39.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:39 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1800752323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:40.784 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:57:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:57:40] "GET /metrics HTTP/1.1" 200 26280 "" "Prometheus/2.43.0" 2024-09-20T18:57:41.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:41 smithi076 ceph-mon[23648]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:41.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:41 smithi076 ceph-mon[30031]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:41.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:41 smithi194 ceph-mon[25770]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:42.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:42.200 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:42.232 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:42.451 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:42.451 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:43.311 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:43 smithi194 ceph-mon[25770]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:43.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:43 smithi076 ceph-mon[23648]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:43.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:43 smithi076 ceph-mon[30031]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:44.047 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:44.050 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:44.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:44 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1928433636' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:44.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:44 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1928433636' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:44.449 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:44 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1928433636' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:45.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:45 smithi076 ceph-mon[23648]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:45.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:45 smithi076 ceph-mon[30031]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:45.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:45 smithi194 ceph-mon[25770]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:47.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[30031]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[23648]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.285 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:46 smithi194 ceph-mon[25770]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:47.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:57:47.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:47.949 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:49.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:48 smithi076 ceph-mon[23648]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:49.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:48 smithi076 ceph-mon[30031]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:49.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:48 smithi194 ceph-mon[25770]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:49.707 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:49.707 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:50.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:49 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2790929434' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:50.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:49 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2790929434' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:50.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:49 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2790929434' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:50.784 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:57:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:57:50] "GET /metrics HTTP/1.1" 200 26281 "" "Prometheus/2.43.0" 2024-09-20T18:57:51.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:50 smithi076 ceph-mon[23648]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:51.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:50 smithi076 ceph-mon[30031]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:51.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:50 smithi194 ceph-mon[25770]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:53.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:52 smithi076 ceph-mon[23648]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:53.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:52 smithi076 ceph-mon[30031]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:53.285 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:53.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:52 smithi194 ceph-mon[25770]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:53.501 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:55.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:54 smithi076 ceph-mon[23648]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:55.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:54 smithi076 ceph-mon[30031]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:55.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:57:55.172 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:57:55.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:54 smithi194 ceph-mon[25770]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:56.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:55 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/616449474' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:56.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:55 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/616449474' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:56.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:55 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/616449474' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:57:57.193 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:56 smithi076 ceph-mon[23648]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:57.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:56 smithi076 ceph-mon[30031]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:57.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:56 smithi194 ceph-mon[25770]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:58.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:58.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:58.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:57:58.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:57:59.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:57:58 smithi076 ceph-mon[23648]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:59.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:57:58 smithi076 ceph-mon[30031]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:57:59.087 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:57:59.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:57:58 smithi194 ceph-mon[25770]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:00.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:58:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:58:00] "GET /metrics HTTP/1.1" 200 26281 "" "Prometheus/2.43.0" 2024-09-20T18:58:00.982 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:00.982 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:01.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:00 smithi076 ceph-mon[23648]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:01.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:00 smithi076 ceph-mon[30031]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:01.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:00 smithi194 ceph-mon[25770]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:01.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:01 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/96790255' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:01.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:01 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/96790255' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:02.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:01 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/96790255' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:03.089 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:02 smithi194 ceph-mon[25770]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:03.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:02 smithi076 ceph-mon[23648]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:03.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:02 smithi076 ceph-mon[30031]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:04.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:04.964 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:05.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:04 smithi076 ceph-mon[23648]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:05.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:04 smithi076 ceph-mon[30031]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:05.089 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:04 smithi194 ceph-mon[25770]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:06.801 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:06.801 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[23648]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3831157822' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[30031]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:07.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:07 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3831157822' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:07.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:07 smithi194 ceph-mon[25770]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:07.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:07.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:07 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3831157822' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:09.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:09 smithi076 ceph-mon[23648]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:09.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:09 smithi076 ceph-mon[30031]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:09.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:09 smithi194 ceph-mon[25770]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:10.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:10.712 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:58:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:58:10] "GET /metrics HTTP/1.1" 200 26281 "" "Prometheus/2.43.0" 2024-09-20T18:58:11.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:11 smithi076 ceph-mon[23648]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:11.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:11 smithi076 ceph-mon[30031]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:11.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:11 smithi194 ceph-mon[25770]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:12.282 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:12.282 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:13.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:13 smithi076 ceph-mon[23648]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:13.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:13 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:13.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:13 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/918171051' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:13.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:13 smithi076 ceph-mon[30031]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:13.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:13 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:13.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:13 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/918171051' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:13.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:13 smithi194 ceph-mon[25770]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:13.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:13 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:13.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:13 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/918171051' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:15.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:15 smithi194 ceph-mon[25770]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:15.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:15 smithi076 ceph-mon[23648]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:15.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:15 smithi076 ceph-mon[30031]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:15.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:17.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:17 smithi194 ceph-mon[25770]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:17.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:17 smithi076 ceph-mon[23648]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:17.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:17 smithi076 ceph-mon[30031]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:17.592 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:19.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:19 smithi076 ceph-mon[23648]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:19.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:19 smithi076 ceph-mon[30031]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:19.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:19 smithi194 ceph-mon[25770]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:20.019 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:20.019 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:20.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:20 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3928543661' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:20.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:20 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3928543661' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:20.342 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:20 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3928543661' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:20.624 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:58:20 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:58:20] "GET /metrics HTTP/1.1" 200 26280 "" "Prometheus/2.43.0" 2024-09-20T18:58:21.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:21 smithi194 ceph-mon[25770]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:21.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:21 smithi076 ceph-mon[30031]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:21.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:21 smithi076 ceph-mon[23648]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:23.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:23 smithi194 ceph-mon[25770]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:23.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:23 smithi076 ceph-mon[23648]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:23.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:23 smithi076 ceph-mon[30031]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:23.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:25.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:25 smithi076 ceph-mon[23648]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:25.220 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:25 smithi076 ceph-mon[30031]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:25.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:25 smithi194 ceph-mon[25770]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:25.619 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:25.619 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:26.163 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:26 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1765962633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:26.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:26 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1765962633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:26.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:26 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1765962633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:27.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[23648]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:27.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:27.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[30031]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:27.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:27.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:27 smithi194 ceph-mon[25770]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:27.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:27.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:29.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:29.495 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:29.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:29 smithi076 ceph-mon[23648]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:29.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:29 smithi076 ceph-mon[30031]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:29.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:29 smithi194 ceph-mon[25770]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:30.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:58:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:58:30] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T18:58:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:31.167 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:31.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:31 smithi076 ceph-mon[23648]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:31.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:31 smithi076 ceph-mon[30031]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:31.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:31 smithi194 ceph-mon[25770]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:32.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:32 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2365850602' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:32.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:32 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2365850602' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:32.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:32 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2365850602' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:33.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:33 smithi076 ceph-mon[23648]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:33.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:33 smithi076 ceph-mon[30031]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:33.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:33 smithi194 ceph-mon[25770]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:34.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:34 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:58:34.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:34 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:58:34.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:34 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:58:34.851 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:35.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:35 smithi076 ceph-mon[30031]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:35.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:35 smithi076 ceph-mon[23648]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:35.501 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:35 smithi194 ceph-mon[25770]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:36.000 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:37.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[30031]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:37.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:58:37.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[23648]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:58:37.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:37 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:37 smithi194 ceph-mon[25770]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:37.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:37 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:37 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:37.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:37 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:58:37.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:37 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:58:37.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:37 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:38.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:38 smithi194 ceph-mon[25770]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:38.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:38 smithi194 ceph-mon[25770]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:38.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:38 smithi076 ceph-mon[23648]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:38.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:38 smithi076 ceph-mon[23648]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:38.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:38 smithi076 ceph-mon[30031]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:38.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:38 smithi076 ceph-mon[30031]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:38.714 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:38.714 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:39.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:39 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2165836037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:39.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:39 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2165836037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:39.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:39 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2165836037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:40.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:40 smithi076 ceph-mon[30031]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:40.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:40 smithi076 ceph-mon[23648]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:40.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:58:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:58:40] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T18:58:40.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:40 smithi194 ceph-mon[25770]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:42.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:42.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:42 smithi194 ceph-mon[25770]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:42.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:42.668 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:42.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:42 smithi076 ceph-mon[23648]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:42.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:42.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:42 smithi076 ceph-mon[30031]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:42.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:44.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:44 smithi076 ceph-mon[23648]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:44.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:44 smithi076 ceph-mon[30031]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:44.549 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:44.549 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:44.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:44 smithi194 ceph-mon[25770]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:45.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:45 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/181770346' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:45.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:45 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/181770346' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:45.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:45 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/181770346' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:46.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:46 smithi194 ceph-mon[25770]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:46.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.589 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:46 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[23648]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:46.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[30031]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:46.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:46.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:46 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:58:47.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:47 smithi076 ceph-mon[23648]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:47.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:47 smithi076 ceph-mon[30031]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:47.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:47 smithi194 ceph-mon[25770]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:48.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:48.411 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:50.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:49 smithi076 ceph-mon[23648]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:50.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:49 smithi076 ceph-mon[30031]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:50.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:49 smithi194 ceph-mon[25770]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:50.155 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:50.155 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:50.778 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:58:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:58:50] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T18:58:51.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:50 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1151055820' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:51.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:50 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1151055820' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:51.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:50 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1151055820' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:52.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:51 smithi076 ceph-mon[23648]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:52.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:51 smithi076 ceph-mon[30031]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:52.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:51 smithi194 ceph-mon[25770]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:53.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:54.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:53 smithi076 ceph-mon[23648]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:54.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:53 smithi076 ceph-mon[30031]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:54.040 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:58:54.089 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:53 smithi194 ceph-mon[25770]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:55.762 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:58:55.762 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:58:56.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:55 smithi076 ceph-mon[23648]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:56.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:55 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1386236609' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:56.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:55 smithi076 ceph-mon[30031]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:56.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:55 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1386236609' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:56.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:55 smithi194 ceph-mon[25770]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:56.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:55 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1386236609' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:58:58.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:57 smithi194 ceph-mon[25770]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:58.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:58.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:57 smithi076 ceph-mon[23648]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:58.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:58.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:57 smithi076 ceph-mon[30031]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:58:58.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:58:59.476 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:58:59.777 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:00.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:58:59 smithi194 ceph-mon[25770]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:00.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:58:59 smithi076 ceph-mon[23648]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:00.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:58:59 smithi076 ceph-mon[30031]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:00.745 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:59:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:59:00] "GET /metrics HTTP/1.1" 200 26277 "" "Prometheus/2.43.0" 2024-09-20T18:59:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:02.221 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:02.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:02 smithi076 ceph-mon[23648]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:02.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:02 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/577555492' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:02.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:02 smithi076 ceph-mon[30031]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:02.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:02 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/577555492' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:02.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:02 smithi194 ceph-mon[25770]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:02.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:02 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/577555492' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:04.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:04 smithi194 ceph-mon[25770]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:04.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:04 smithi076 ceph-mon[23648]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:04.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:04 smithi076 ceph-mon[30031]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:05.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:06.209 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:06.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:06 smithi194 ceph-mon[25770]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:06.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:06 smithi076 ceph-mon[23648]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:06.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:06 smithi076 ceph-mon[30031]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:07.666 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[23648]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:07 smithi076 ceph-mon[30031]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:07.758 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.758 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.758 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:07.758 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:07 smithi194 ceph-mon[25770]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:08.053 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:08.053 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:08.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:08 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1669759154' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:08.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:08 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1669759154' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:08.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:08 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1669759154' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:09.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:09 smithi076 ceph-mon[23648]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:09.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:09 smithi076 ceph-mon[30031]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:09.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:09 smithi194 ceph-mon[25770]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:10.783 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:59:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:59:10] "GET /metrics HTTP/1.1" 200 26277 "" "Prometheus/2.43.0" 2024-09-20T18:59:11.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:11.874 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:11.994 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:11 smithi076 ceph-mon[23648]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:11.994 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:11 smithi076 ceph-mon[30031]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:12.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:11 smithi194 ceph-mon[25770]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:13.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:12 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:13.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:12 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:13.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:12 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:13.683 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:13.683 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:14.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:13 smithi076 ceph-mon[30031]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:14.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:13 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/978383488' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:14.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:13 smithi076 ceph-mon[23648]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:14.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:13 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/978383488' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:14.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:13 smithi194 ceph-mon[25770]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:14.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:13 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/978383488' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:16.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:15 smithi194 ceph-mon[25770]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:16.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:15 smithi076 ceph-mon[23648]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:16.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:15 smithi076 ceph-mon[30031]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:17.412 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:17.619 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:18.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:17 smithi076 ceph-mon[23648]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:18.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:17 smithi076 ceph-mon[30031]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:18.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:17 smithi194 ceph-mon[25770]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:20.007 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:19 smithi076 ceph-mon[23648]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:20.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:19 smithi076 ceph-mon[30031]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:20.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:19 smithi194 ceph-mon[25770]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:20.657 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:59:20 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:59:20] "GET /metrics HTTP/1.1" 200 26275 "" "Prometheus/2.43.0" 2024-09-20T18:59:21.083 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:21.083 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:22.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:21 smithi076 ceph-mon[30031]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:22.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:21 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2348371101' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:22.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:21 smithi076 ceph-mon[23648]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:22.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:21 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2348371101' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:22.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:21 smithi194 ceph-mon[25770]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:22.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:21 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2348371101' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:24.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:23 smithi194 ceph-mon[25770]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:24.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:23 smithi076 ceph-mon[23648]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:24.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:23 smithi076 ceph-mon[30031]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:25.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:25.284 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:26.010 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:25 smithi194 ceph-mon[25770]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:26.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:25 smithi076 ceph-mon[23648]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:26.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:25 smithi076 ceph-mon[30031]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:27.148 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:27.148 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:27.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[30031]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2336398668' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[23648]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:27.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:27 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2336398668' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:27.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:27.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:27 smithi194 ceph-mon[25770]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:27.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:27.838 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:27 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2336398668' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:30.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:29 smithi076 ceph-mon[30031]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:30.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:29 smithi076 ceph-mon[23648]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:30.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:29 smithi194 ceph-mon[25770]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:30.782 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:59:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:59:30] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T18:59:30.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:31.043 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:32.087 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:31 smithi076 ceph-mon[23648]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:32.087 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:31 smithi076 ceph-mon[30031]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:32.087 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:31 smithi194 ceph-mon[25770]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:32.797 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:34.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:33 smithi194 ceph-mon[25770]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:34.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:33 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3837803267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:34.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:33 smithi076 ceph-mon[23648]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:34.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:33 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3837803267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:34.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:33 smithi076 ceph-mon[30031]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:34.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:33 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3837803267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:36.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:35 smithi194 ceph-mon[25770]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:36.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:35 smithi076 ceph-mon[23648]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:36.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:35 smithi076 ceph-mon[30031]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:36.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:36.575 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:38.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:37 smithi076 ceph-mon[23648]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:38.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:37 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:59:38.065 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:37 smithi076 ceph-mon[30031]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:38.065 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:37 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:59:38.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:37 smithi194 ceph-mon[25770]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:38.088 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:37 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T18:59:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:38.364 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:39.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:38 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/4290114004' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:39.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:38 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/4290114004' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:39.087 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:38 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/4290114004' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:40.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:39 smithi076 ceph-mon[23648]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:40.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:39 smithi076 ceph-mon[30031]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:40.087 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:39 smithi194 ceph-mon[25770]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:40.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:59:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:59:40] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T18:59:42.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:41 smithi076 ceph-mon[23648]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:42.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:41 smithi076 ceph-mon[30031]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:42.087 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:41 smithi194 ceph-mon[25770]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:42.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:42.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:42.946 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:43.087 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:43.569 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:43 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:44.338 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:43 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:45.399 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:45.399 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:46.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:46 smithi194 ceph-mon[25770]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:46.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:46 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1061079005' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:46.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:46 smithi076 ceph-mon[23648]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:46.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:46 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1061079005' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:46.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:46 smithi076 ceph-mon[30031]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:46.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:46 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1061079005' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:47.762 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:47 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:47.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:47 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:47.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:47 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:47.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:47 smithi194 ceph-mon[25770]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:48.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:48.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:48.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:48.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[23648]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:48.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:48.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:48.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T18:59:48.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:47 smithi076 ceph-mon[30031]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:49.102 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:49.313 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:50.588 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:50 smithi194 ceph-mon[25770]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:50.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:50 smithi076 ceph-mon[23648]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:50.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:50 smithi076 ceph-mon[30031]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:50.647 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 18:59:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:18:59:50] "GET /metrics HTTP/1.1" 200 26279 "" "Prometheus/2.43.0" 2024-09-20T18:59:51.214 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:51.215 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:51.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:51 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/992995785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:51.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:51 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/992995785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:51.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:51 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/992995785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:52.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:52 smithi194 ceph-mon[25770]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:52.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:52 smithi076 ceph-mon[23648]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:52.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:52 smithi076 ceph-mon[30031]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:54.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:54 smithi194 ceph-mon[25770]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:54.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:54 smithi076 ceph-mon[23648]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:54.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:54 smithi076 ceph-mon[30031]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:54.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T18:59:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T18:59:56.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:56 smithi076 ceph-mon[23648]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:56.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:56 smithi076 ceph-mon[30031]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:56.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:56 smithi194 ceph-mon[25770]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:56.804 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T18:59:56.804 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T18:59:57.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:57 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/741335656' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:57.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:57 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:57 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/741335656' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:57 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:57.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:57 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/741335656' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T18:59:57.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:57 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T18:59:58.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 18:59:58 smithi194 ceph-mon[25770]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:58.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 18:59:58 smithi076 ceph-mon[23648]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T18:59:58.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 18:59:58 smithi076 ceph-mon[30031]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:00.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:00.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:00 smithi194 ceph-mon[25770]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:00.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:00 smithi194 ceph-mon[25770]: Health detail: HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:00:00.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:00 smithi194 ceph-mon[25770]: [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:00:00.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:00 smithi194 ceph-mon[25770]: Cephadm agent on host smithi076 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:00:00.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[23648]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:00.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[23648]: Health detail: HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:00:00.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[23648]: [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:00:00.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[23648]: Cephadm agent on host smithi076 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:00:00.589 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:00:00] "GET /metrics HTTP/1.1" 200 26270 "" "Prometheus/2.43.0" 2024-09-20T19:00:00.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[30031]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:00.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[30031]: Health detail: HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:00:00.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[30031]: [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:00:00.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:00 smithi076 ceph-mon[30031]: Cephadm agent on host smithi076 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:00:01.736 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:02.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:02 smithi076 ceph-mon[30031]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:02.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:02 smithi076 ceph-mon[23648]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:02.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:02 smithi194 ceph-mon[25770]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:04.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:04 smithi076 ceph-mon[23648]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:04.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:04 smithi076 ceph-mon[30031]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:04.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:04 smithi194 ceph-mon[25770]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:04.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:04.760 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:05.431 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:05 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/126711847' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:05.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:05 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/126711847' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:05.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:05 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/126711847' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:06.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:06 smithi194 ceph-mon[25770]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:06.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:06 smithi076 ceph-mon[23648]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:06.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:06 smithi076 ceph-mon[30031]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:07.963 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:07.963 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:07.963 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:07 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:07.964 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:07 smithi194 ceph-mon[25770]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[23648]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:08.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:07 smithi076 ceph-mon[30031]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:08.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:08.675 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:10.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:10 smithi076 ceph-mon[23648]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:10.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:10 smithi076 ceph-mon[30031]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:10.500 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:10.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:10 smithi194 ceph-mon[25770]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:10.781 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:00:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:00:10] "GET /metrics HTTP/1.1" 200 26270 "" "Prometheus/2.43.0" 2024-09-20T19:00:11.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:11 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2359184408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:11.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:11 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2359184408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:11.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:11 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2359184408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:12.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:12 smithi194 ceph-mon[25770]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:12.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:12 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:12.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:12 smithi076 ceph-mon[23648]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:12.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:12 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:12.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:12 smithi076 ceph-mon[30031]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:12.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:12 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:14.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:14.324 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:14.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:14 smithi076 ceph-mon[23648]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:14.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:14 smithi076 ceph-mon[30031]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:14.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:14 smithi194 ceph-mon[25770]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:16.179 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:16.179 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:16.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:16 smithi076 ceph-mon[23648]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:16.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:16 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1193421' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:16.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:16 smithi076 ceph-mon[30031]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:16.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:16 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1193421' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:16.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:16 smithi194 ceph-mon[25770]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:16.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:16 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1193421' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:18.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:18 smithi194 ceph-mon[25770]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:18.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:18 smithi076 ceph-mon[23648]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:18.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:18 smithi076 ceph-mon[30031]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:19.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:20.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:20 smithi076 ceph-mon[23648]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:20.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:00:20 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:00:20] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T19:00:20.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:20 smithi076 ceph-mon[30031]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:20.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:20 smithi194 ceph-mon[25770]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:21.251 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:22.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:22 smithi194 ceph-mon[25770]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:22.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:22 smithi076 ceph-mon[23648]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:22.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:22 smithi076 ceph-mon[30031]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:24.220 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:24.220 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:24.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:24 smithi076 ceph-mon[23648]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:24.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:24 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1842222364' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:24.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:24 smithi076 ceph-mon[30031]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:24.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:24 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1842222364' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:24.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:24 smithi194 ceph-mon[25770]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:24.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:24 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1842222364' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:26.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:26 smithi194 ceph-mon[25770]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:26.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:26 smithi076 ceph-mon[30031]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:26.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:26 smithi076 ceph-mon[23648]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:27.894 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:28.111 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:28.232 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.232 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[30031]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:28.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:27 smithi076 ceph-mon[23648]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:28.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:28.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:27 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:28.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:27 smithi194 ceph-mon[25770]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:29.871 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:29.872 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:30.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:30 smithi076 ceph-mon[23648]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:30.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:30 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2056188019' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:30.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:00:30 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:00:30] "GET /metrics HTTP/1.1" 200 26277 "" "Prometheus/2.43.0" 2024-09-20T19:00:30.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:30 smithi076 ceph-mon[30031]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:30.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:30 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2056188019' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:30.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:30 smithi194 ceph-mon[25770]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:30.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:30 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2056188019' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:32.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:32 smithi076 ceph-mon[23648]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:32.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:32 smithi076 ceph-mon[30031]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:32.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:32 smithi194 ceph-mon[25770]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:33.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:34.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:34 smithi076 ceph-mon[23648]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:34.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:34 smithi076 ceph-mon[30031]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:34.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:34 smithi194 ceph-mon[25770]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:35.611 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:35.612 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:36.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:36 smithi194 ceph-mon[25770]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:36.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:36 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1443422091' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:36.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:36 smithi076 ceph-mon[23648]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:36.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:36 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1443422091' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:36.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:36 smithi076 ceph-mon[30031]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:36.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:36 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1443422091' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:38.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:38 smithi194 ceph-mon[25770]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:38.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:38 smithi076 ceph-mon[23648]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:38.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:38 smithi076 ceph-mon[30031]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:39.339 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:39.552 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:40.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:40 smithi076 ceph-mon[23648]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:40.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:40 smithi076 ceph-mon[30031]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:40.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:00:40 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:00:40] "GET /metrics HTTP/1.1" 200 26277 "" "Prometheus/2.43.0" 2024-09-20T19:00:40.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:40 smithi194 ceph-mon[25770]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:42.435 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:42.435 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:42.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:42 smithi194 ceph-mon[25770]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:42.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:42 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:42.751 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:42 smithi076 ceph-mon[23648]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:42.752 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:42 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:42.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:42 smithi076 ceph-mon[30031]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:42.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:42 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:43.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:43 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3233388' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:43.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:43 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3233388' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:43.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:43 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3233388' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:44.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:44 smithi194 ceph-mon[25770]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:44.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:00:44.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:44 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[23648]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:44.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:00:44.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[30031]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:44.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:00:44.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:44.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:44 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:45.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:45 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:00:45.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:45 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:00:45.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:45 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:45.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:45 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:00:45.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:45 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:00:45.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:45 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:45.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:45 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:00:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:45 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:00:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:45 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:46.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:46.387 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:46.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:46 smithi076 ceph-mon[30031]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:46.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:46 smithi076 ceph-mon[30031]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:46.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:46 smithi076 ceph-mon[23648]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:46.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:46 smithi076 ceph-mon[23648]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:46.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:46 smithi194 ceph-mon[25770]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:46.587 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:46 smithi194 ceph-mon[25770]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:48.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[23648]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:48.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:48.170 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:48.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[30031]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:48.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:48 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:48 smithi194 ceph-mon[25770]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:48.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:48 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:48 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:48.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:48 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:00:49.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:49 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1042344795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:49.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:49 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1042344795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:49.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:49 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1042344795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:50.337 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:50 smithi194 ceph-mon[25770]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:50.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:50 smithi076 ceph-mon[23648]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:50.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:50 smithi076 ceph-mon[30031]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:50.781 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:00:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:00:50] "GET /metrics HTTP/1.1" 200 26276 "" "Prometheus/2.43.0" 2024-09-20T19:00:51.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:52.188 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:52.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:52 smithi076 ceph-mon[23648]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:52.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:52 smithi076 ceph-mon[30031]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:52.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:52 smithi194 ceph-mon[25770]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:54.088 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:54.088 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:00:54.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:54 smithi194 ceph-mon[25770]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:54.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:54 smithi076 ceph-mon[23648]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:54.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:54 smithi076 ceph-mon[30031]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:55.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:55 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/1255374223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:55.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:55 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/1255374223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:55 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/1255374223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:00:56.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:56 smithi194 ceph-mon[25770]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:56.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:56 smithi076 ceph-mon[23648]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:56.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:56 smithi076 ceph-mon[30031]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:57.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:00:57.892 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:00:58.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:58 smithi076 ceph-mon[23648]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:58.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:00:58 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:58.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:58 smithi076 ceph-mon[30031]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:58.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:00:58 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:58.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:58 smithi194 ceph-mon[25770]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:00:58.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:00:58 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:00:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:00:59.818 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:01:00.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:00 smithi076 ceph-mon[23648]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:00.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:00 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/2326684537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:00.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:00 smithi076 ceph-mon[30031]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:00.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:00 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/2326684537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:00.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:00 smithi194 ceph-mon[25770]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:00.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:00 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/2326684537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:00.781 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:00 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:01:00] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T19:01:02.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:02 smithi076 ceph-mon[23648]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:02.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:02 smithi076 ceph-mon[30031]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:02.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:02 smithi194 ceph-mon[25770]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:03.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:01:04.292 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:01:04.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:04 smithi194 ceph-mon[25770]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:04.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:04 smithi076 ceph-mon[23648]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:04.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:04 smithi076 ceph-mon[30031]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:01:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:01:06.336 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:06 smithi194 ceph-mon[25770]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:06.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:06 smithi076 ceph-mon[23648]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:06.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:06 smithi076 ceph-mon[30031]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:07.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:07 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3536413369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:07.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:07 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3536413369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:07.324 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:07 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3536413369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:08.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[23648]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:08.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[30031]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:08.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:08 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:08 smithi194 ceph-mon[25770]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:08.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:08 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:08 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:08.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:08 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' 2024-09-20T19:01:09.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid eefb3510-7780-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:01:10.062 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:01:10.310 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:10 smithi194 ceph-mon[25770]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:10.423 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:10 smithi076 ceph-mon[23648]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:10.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:10 smithi076 ceph-mon[30031]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:10.781 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:10 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: ::ffff:172.21.15.194 - - [20/Sep/2024:19:01:10] "GET /metrics HTTP/1.1" 200 26278 "" "Prometheus/2.43.0" 2024-09-20T19:01:11.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-20T19:01:11.931 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]} 2024-09-20T19:01:12.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:12 smithi076 ceph-mon[23648]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:12.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:12 smithi076 ceph-mon[23648]: from='client.? 172.21.15.76:0/3051804967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:12.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:12 smithi076 ceph-mon[23648]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:01:12.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:12 smithi076 ceph-mon[30031]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:12.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:12 smithi076 ceph-mon[30031]: from='client.? 172.21.15.76:0/3051804967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:12.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:12 smithi076 ceph-mon[30031]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:01:12.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:12 smithi194 ceph-mon[25770]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:01:12.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:12 smithi194 ceph-mon[25770]: from='client.? 172.21.15.76:0/3051804967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:01:12.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:12 smithi194 ceph-mon[25770]: from='mgr.14445 172.21.15.76:0/1153011165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:01:12.655 INFO:tasks.cephadm:Teardown begin 2024-09-20T19:01:12.655 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/cephadm.py", line 2371, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/ceph.py", line 1537, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/ceph_manager.py", line 3303, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-09-20T19:01:12.658 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:01:12.685 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:01:12.721 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-20T19:01:12.721 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:01:12.740 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:01:12.778 INFO:tasks.cephadm:Stopping all daemons... 2024-09-20T19:01:12.779 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-20T19:01:12.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a 2024-09-20T19:01:13.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:12 smithi076 systemd[1]: Stopping Ceph mon.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:13.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a[23644]: 2024-09-20T19:01:13.079+0000 7f1796a77640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:13.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a[23644]: 2024-09-20T19:01:13.079+0000 7f1796a77640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T19:01:13.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:13 smithi076 podman[73416]: 2024-09-20 19:01:13.099854133 +0000 UTC m=+0.143866602 container died 856419dcdf1284f6960ad3b9b84af688230c38c666727a5c3187907570638b00 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T19:01:13.371 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:13] ENGINE Bus STOPPING 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:13 smithi076 podman[73416]: 2024-09-20 19:01:13.453848401 +0000 UTC m=+0.497860872 container cleanup 856419dcdf1284f6960ad3b9b84af688230c38c666727a5c3187907570638b00 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Sep 20 19:01:13 smithi076 bash[73416]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-a 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:13] ENGINE Bus STOPPED 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:13] ENGINE Bus STARTING 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:13] ENGINE Serving on http://:::9283 2024-09-20T19:01:13.721 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:13 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:13] ENGINE Bus STARTED 2024-09-20T19:01:13.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.a.service' 2024-09-20T19:01:13.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:13.977 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-20T19:01:13.977 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-20T19:01:13.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.c 2024-09-20T19:01:14.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:14 smithi076 systemd[1]: Stopping Ceph mon.c for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:14.543 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:14] ENGINE Bus STOPPING 2024-09-20T19:01:14.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c[30027]: 2024-09-20T19:01:14.282+0000 7ff180189640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:14.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c[30027]: 2024-09-20T19:01:14.282+0000 7ff180189640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T19:01:14.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:14 smithi076 podman[73547]: 2024-09-20 19:01:14.416551637 +0000 UTC m=+0.263144820 container died 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:01:14.857 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:01:14.857 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:14] ENGINE Bus STOPPED 2024-09-20T19:01:14.858 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:14] ENGINE Bus STARTING 2024-09-20T19:01:14.858 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:14] ENGINE Serving on http://:::9283 2024-09-20T19:01:14.858 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:14 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y[23877]: [20/Sep/2024:19:01:14] ENGINE Bus STARTED 2024-09-20T19:01:15.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:14 smithi076 podman[73547]: 2024-09-20 19:01:14.850038319 +0000 UTC m=+0.696631500 container cleanup 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-20T19:01:15.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:14 smithi076 bash[73547]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c 2024-09-20T19:01:15.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Sep 20 19:01:15 smithi076 podman[73559]: 2024-09-20 19:01:15.042887624 +0000 UTC m=+0.623176438 container remove 43d0a1e864dda9d4255480518f39d6b0c09104df5ae6811c448febe19d3368e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-c, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:01:15.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.c.service' 2024-09-20T19:01:15.322 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:15.323 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-20T19:01:15.323 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-20T19:01:15.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.b 2024-09-20T19:01:15.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:15 smithi194 systemd[1]: Stopping Ceph mon.b for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:15.917 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:15 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b[25755]: 2024-09-20T19:01:15.617+0000 7f9bfcb29640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:15.918 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:15 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b[25755]: 2024-09-20T19:01:15.617+0000 7f9bfcb29640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T19:01:15.918 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:15 smithi194 podman[53512]: 2024-09-20 19:01:15.638263036 +0000 UTC m=+0.128457462 container died 0bafb7d000b40b3071e6343e92ee65bd80a35c6d1fdb7327fdfe9d9b47eea2dd (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-20T19:01:16.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:16 smithi194 podman[53512]: 2024-09-20 19:01:16.009103152 +0000 UTC m=+0.499297623 container cleanup 0bafb7d000b40b3071e6343e92ee65bd80a35c6d1fdb7327fdfe9d9b47eea2dd (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:01:16.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 20 19:01:16 smithi194 bash[53512]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-mon-b 2024-09-20T19:01:16.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mon.b.service' 2024-09-20T19:01:16.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:16.493 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-20T19:01:16.493 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-20T19:01:16.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y 2024-09-20T19:01:16.820 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:16 smithi076 systemd[1]: Stopping Ceph mgr.y for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:17.079 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:16 smithi076 podman[73680]: 2024-09-20 19:01:16.818169073 +0000 UTC m=+0.143304721 container died db9f4b4ab948694b60f2f7dc5f1e8a7d59b54d6e8dda945e5a4f579cc9213668 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T19:01:17.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:17 smithi076 podman[73680]: 2024-09-20 19:01:17.279244188 +0000 UTC m=+0.604379878 container cleanup db9f4b4ab948694b60f2f7dc5f1e8a7d59b54d6e8dda945e5a4f579cc9213668 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:01:17.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Sep 20 19:01:17 smithi076 bash[73680]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-y 2024-09-20T19:01:17.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-20T19:01:17.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:17.738 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-20T19:01:17.738 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-20T19:01:17.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.x 2024-09-20T19:01:18.050 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 19:01:17 smithi194 systemd[1]: Stopping Ceph mgr.x for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:18.336 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 19:01:18 smithi194 podman[53634]: 2024-09-20 19:01:18.043298152 +0000 UTC m=+0.117654716 container died 7b8f27eb749782df3f2b1852d9b98388c059af652648324dc3c78f583ee49c04 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:01:18.689 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 19:01:18 smithi194 podman[53634]: 2024-09-20 19:01:18.488872986 +0000 UTC m=+0.563229550 container cleanup 7b8f27eb749782df3f2b1852d9b98388c059af652648324dc3c78f583ee49c04 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:01:18.689 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 20 19:01:18 smithi194 bash[53634]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-mgr-x 2024-09-20T19:01:18.910 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-20T19:01:18.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:18.946 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-20T19:01:18.946 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-20T19:01:18.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.0 2024-09-20T19:01:19.259 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:18 smithi076 systemd[1]: Stopping Ceph osd.0 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:19.531 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:19 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0[34318]: 2024-09-20T19:01:19.250+0000 7f5127ad4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:19.531 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:19 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0[34318]: 2024-09-20T19:01:19.250+0000 7f5127ad4640 -1 osd.0 59 *** Got signal Terminated *** 2024-09-20T19:01:19.531 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:19 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0[34318]: 2024-09-20T19:01:19.250+0000 7f5127ad4640 -1 osd.0 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:01:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:01:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:01:22.897Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:01:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:01:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:01:22.897Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:01:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:01:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:01:22.900Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:01:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:01:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:01:22.900Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:01:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:01:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:01:22.900Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:01:25.676 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:25 smithi076 podman[73825]: 2024-09-20 19:01:25.407594719 +0000 UTC m=+6.273946316 container died a1f436cf143a3c3913470515477dace5840d3b678b0df4a08dd492b6cae73e13 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T19:01:26.002 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:25 smithi076 podman[73825]: 2024-09-20 19:01:25.835213619 +0000 UTC m=+6.701565206 container cleanup a1f436cf143a3c3913470515477dace5840d3b678b0df4a08dd492b6cae73e13 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-0b2cc4d) 2024-09-20T19:01:26.002 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:25 smithi076 bash[73825]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:01:26.280 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:25 smithi076 podman[74251]: 2024-09-20 19:01:25.995038025 +0000 UTC m=+1.213136447 container remove a1f436cf143a3c3913470515477dace5840d3b678b0df4a08dd492b6cae73e13 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T19:01:28.130 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:27 smithi076 podman[74543]: 2024-09-20 19:01:27.774134192 +0000 UTC m=+0.016438857 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:01:28.130 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:27 smithi076 podman[74543]: 2024-09-20 19:01:27.878421128 +0000 UTC m=+0.120725774 container create a0cd9676ffb0c6dab4b6db19909ec6d7ca7339196a2140ddaf00acb88ddadbac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T19:01:28.130 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:28 smithi076 podman[74543]: 2024-09-20 19:01:28.122636397 +0000 UTC m=+0.364941060 container init a0cd9676ffb0c6dab4b6db19909ec6d7ca7339196a2140ddaf00acb88ddadbac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T19:01:28.531 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:28 smithi076 podman[74543]: 2024-09-20 19:01:28.126308772 +0000 UTC m=+0.368613421 container start a0cd9676ffb0c6dab4b6db19909ec6d7ca7339196a2140ddaf00acb88ddadbac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T19:01:28.531 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:28 smithi076 podman[74543]: 2024-09-20 19:01:28.205867058 +0000 UTC m=+0.448171723 container attach a0cd9676ffb0c6dab4b6db19909ec6d7ca7339196a2140ddaf00acb88ddadbac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T19:01:28.531 INFO:journalctl@ceph.osd.0.smithi076.stdout:Sep 20 19:01:28 smithi076 podman[74543]: 2024-09-20 19:01:28.311655949 +0000 UTC m=+0.553960610 container died a0cd9676ffb0c6dab4b6db19909ec6d7ca7339196a2140ddaf00acb88ddadbac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True) 2024-09-20T19:01:29.062 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.0.service' 2024-09-20T19:01:29.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:29.103 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-20T19:01:29.103 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-20T19:01:29.104 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.1 2024-09-20T19:01:29.420 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:29 smithi076 systemd[1]: Stopping Ceph osd.1 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:29.780 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:29 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1[37802]: 2024-09-20T19:01:29.412+0000 7f6864911640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:29.780 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:29 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1[37802]: 2024-09-20T19:01:29.412+0000 7f6864911640 -1 osd.1 59 *** Got signal Terminated *** 2024-09-20T19:01:29.781 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:29 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1[37802]: 2024-09-20T19:01:29.412+0000 7f6864911640 -1 osd.1 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:01:35.062 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:34 smithi076 podman[74659]: 2024-09-20 19:01:34.794587946 +0000 UTC m=+5.479213817 container died 3375a7473c28968fd216c07e67490890e4a79f016c36bee555f6c3bceb179763 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:01:35.530 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:35 smithi076 podman[74659]: 2024-09-20 19:01:35.237002255 +0000 UTC m=+5.921628117 container cleanup 3375a7473c28968fd216c07e67490890e4a79f016c36bee555f6c3bceb179763 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1, ceph=True, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:01:35.531 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:35 smithi076 bash[74659]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1 2024-09-20T19:01:35.863 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:35 smithi076 podman[74675]: 2024-09-20 19:01:35.530020345 +0000 UTC m=+0.737605733 container remove 3375a7473c28968fd216c07e67490890e4a79f016c36bee555f6c3bceb179763 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T19:01:36.149 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:35 smithi076 podman[74764]: 2024-09-20 19:01:35.855965956 +0000 UTC m=+0.110309546 container create 26066cd0fc3b673f1aaabe7cec1ffa33af2de3cf22b4ee492e0f65caa29d36c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d) 2024-09-20T19:01:36.149 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:35 smithi076 podman[74764]: 2024-09-20 19:01:35.761692305 +0000 UTC m=+0.016035902 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:01:36.149 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:36 smithi076 podman[74764]: 2024-09-20 19:01:36.066843441 +0000 UTC m=+0.321187038 container init 26066cd0fc3b673f1aaabe7cec1ffa33af2de3cf22b4ee492e0f65caa29d36c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T19:01:36.149 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:36 smithi076 podman[74764]: 2024-09-20 19:01:36.070306976 +0000 UTC m=+0.324650564 container start 26066cd0fc3b673f1aaabe7cec1ffa33af2de3cf22b4ee492e0f65caa29d36c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:01:36.149 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:36 smithi076 podman[74764]: 2024-09-20 19:01:36.141801243 +0000 UTC m=+0.396144835 container attach 26066cd0fc3b673f1aaabe7cec1ffa33af2de3cf22b4ee492e0f65caa29d36c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T19:01:36.507 INFO:journalctl@ceph.osd.1.smithi076.stdout:Sep 20 19:01:36 smithi076 podman[74764]: 2024-09-20 19:01:36.259237856 +0000 UTC m=+0.513581453 container died 26066cd0fc3b673f1aaabe7cec1ffa33af2de3cf22b4ee492e0f65caa29d36c4 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:01:36.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.1.service' 2024-09-20T19:01:37.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:37.013 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-20T19:01:37.014 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-20T19:01:37.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.2 2024-09-20T19:01:37.280 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:37 smithi076 systemd[1]: Stopping Ceph osd.2 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:37.780 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2[41303]: 2024-09-20T19:01:37.355+0000 7efff6499640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:37.781 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2[41303]: 2024-09-20T19:01:37.355+0000 7efff6499640 -1 osd.2 59 *** Got signal Terminated *** 2024-09-20T19:01:37.781 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:37 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2[41303]: 2024-09-20T19:01:37.355+0000 7efff6499640 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:01:43.031 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:42 smithi076 podman[74877]: 2024-09-20 19:01:42.928414853 +0000 UTC m=+5.693499672 container died da9ecc540396d514d6944cf03ed47ad0578250f75c23d5804e77aa7690ac7167 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-20T19:01:43.530 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:43 smithi076 podman[74877]: 2024-09-20 19:01:43.283885414 +0000 UTC m=+6.048970230 container cleanup da9ecc540396d514d6944cf03ed47ad0578250f75c23d5804e77aa7690ac7167 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:01:43.530 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:43 smithi076 bash[74877]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2 2024-09-20T19:01:43.961 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:43 smithi076 podman[75003]: 2024-09-20 19:01:43.560167376 +0000 UTC m=+0.678583684 container remove da9ecc540396d514d6944cf03ed47ad0578250f75c23d5804e77aa7690ac7167 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d) 2024-09-20T19:01:44.606 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:44 smithi076 podman[75093]: 2024-09-20 19:01:44.329883047 +0000 UTC m=+0.102254210 container create 5b35ddfc7d31c5b38612494b46a69df96e97bf28a8e1f39c9747cca0a880974e (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-20T19:01:44.606 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:44 smithi076 podman[75093]: 2024-09-20 19:01:44.243391459 +0000 UTC m=+0.015762628 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:01:44.606 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:44 smithi076 podman[75093]: 2024-09-20 19:01:44.524299867 +0000 UTC m=+0.296671037 container init 5b35ddfc7d31c5b38612494b46a69df96e97bf28a8e1f39c9747cca0a880974e (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD) 2024-09-20T19:01:44.607 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:44 smithi076 podman[75093]: 2024-09-20 19:01:44.527926846 +0000 UTC m=+0.300298013 container start 5b35ddfc7d31c5b38612494b46a69df96e97bf28a8e1f39c9747cca0a880974e (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T19:01:44.607 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:44 smithi076 podman[75093]: 2024-09-20 19:01:44.599096367 +0000 UTC m=+0.371467533 container attach 5b35ddfc7d31c5b38612494b46a69df96e97bf28a8e1f39c9747cca0a880974e (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:01:45.030 INFO:journalctl@ceph.osd.2.smithi076.stdout:Sep 20 19:01:44 smithi076 podman[75093]: 2024-09-20 19:01:44.712610297 +0000 UTC m=+0.484981461 container died 5b35ddfc7d31c5b38612494b46a69df96e97bf28a8e1f39c9747cca0a880974e (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d) 2024-09-20T19:01:45.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.2.service' 2024-09-20T19:01:45.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:45.413 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-20T19:01:45.413 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T19:01:45.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.3 2024-09-20T19:01:45.738 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:45 smithi076 systemd[1]: Stopping Ceph osd.3 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:46.030 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:45 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T19:01:45.730+0000 7fd21c037640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:46.030 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:45 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T19:01:45.730+0000 7fd21c037640 -1 osd.3 59 *** Got signal Terminated *** 2024-09-20T19:01:46.031 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:45 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T19:01:45.730+0000 7fd21c037640 -1 osd.3 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:01:46.836 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:46 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:46.784+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:46.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:46 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:46.557+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:47.106 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:46 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:46.845+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:47.862 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:47 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:47.758+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:47.862 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:47 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:47.589+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:48.335 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:47 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:47.854+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:48.780 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:48 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T19:01:48.506+0000 7fd21864b640 -1 osd.3 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.323120+0000 front 2024-09-20T19:01:23.323026+0000 (oldest deadline 2024-09-20T19:01:48.023044+0000) 2024-09-20T19:01:49.085 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:48 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:48.765+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:49.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:48 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:48.825+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:49.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:48 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:48.581+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:49.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:48 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:48.772+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:50.030 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:49 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T19:01:49.549+0000 7fd21864b640 -1 osd.3 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.323120+0000 front 2024-09-20T19:01:23.323026+0000 (oldest deadline 2024-09-20T19:01:48.023044+0000) 2024-09-20T19:01:50.086 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:49 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:49.748+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:50.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:49 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:49.807+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:50.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:49 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:49.839+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:50.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:49 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:49.611+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:50.745 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:50 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:50.644+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:51.030 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:50 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3[44903]: 2024-09-20T19:01:50.556+0000 7fd21864b640 -1 osd.3 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.323120+0000 front 2024-09-20T19:01:23.323026+0000 (oldest deadline 2024-09-20T19:01:48.023044+0000) 2024-09-20T19:01:51.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:50 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:50.818+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:51.086 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:50 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:50.738+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:51.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:50 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:50.796+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:51.319 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:51 smithi076 podman[75195]: 2024-09-20 19:01:51.046919796 +0000 UTC m=+5.416040296 container died 7ae117a641453ac58856ae29f0c58ebbdb7539f4db6a1eed595e4ddb903ec857 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d) 2024-09-20T19:01:51.711 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:51 smithi076 podman[75195]: 2024-09-20 19:01:51.486498389 +0000 UTC m=+5.855618933 container cleanup 7ae117a641453ac58856ae29f0c58ebbdb7539f4db6a1eed595e4ddb903ec857 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:01:51.711 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:51 smithi076 bash[75195]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3 2024-09-20T19:01:51.723 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:51 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:51.678+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:52.030 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:51 smithi076 podman[75761]: 2024-09-20 19:01:51.70435818 +0000 UTC m=+0.653043198 container remove 7ae117a641453ac58856ae29f0c58ebbdb7539f4db6a1eed595e4ddb903ec857 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-20T19:01:52.085 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:51 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:51.715+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:52.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:51 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:51.855+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:52.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:51 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:51.757+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:52.340 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75854]: 2024-09-20 19:01:51.930265004 +0000 UTC m=+0.015798450 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:01:52.340 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75854]: 2024-09-20 19:01:52.046696618 +0000 UTC m=+0.132230054 container create db32dbd975fb781d7b17e3097fcb3bc94736bb4efe101d345a5b3507df7610a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T19:01:52.340 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75854]: 2024-09-20 19:01:52.249251038 +0000 UTC m=+0.334784479 container init db32dbd975fb781d7b17e3097fcb3bc94736bb4efe101d345a5b3507df7610a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T19:01:52.340 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75854]: 2024-09-20 19:01:52.25247131 +0000 UTC m=+0.338004746 container start db32dbd975fb781d7b17e3097fcb3bc94736bb4efe101d345a5b3507df7610a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T19:01:52.340 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75854]: 2024-09-20 19:01:52.3325321 +0000 UTC m=+0.418065551 container attach db32dbd975fb781d7b17e3097fcb3bc94736bb4efe101d345a5b3507df7610a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:01:52.734 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75854]: 2024-09-20 19:01:52.439080855 +0000 UTC m=+0.524614292 container died db32dbd975fb781d7b17e3097fcb3bc94736bb4efe101d345a5b3507df7610a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-0b2cc4d) 2024-09-20T19:01:52.986 INFO:journalctl@ceph.osd.3.smithi076.stdout:Sep 20 19:01:52 smithi076 podman[75877]: 2024-09-20 19:01:52.977144566 +0000 UTC m=+0.532365714 container remove db32dbd975fb781d7b17e3097fcb3bc94736bb4efe101d345a5b3507df7610a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T19:01:52.987 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:52 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:52.678+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:52.987 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:52 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:52.818+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:52.988 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:52 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:52.747+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:52.988 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:52 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:52.667+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:53.005 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T19:01:53.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:01:53.042 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T19:01:53.043 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-20T19:01:53.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.4 2024-09-20T19:01:53.335 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:53 smithi194 systemd[1]: Stopping Ceph osd.4 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:01:53.634 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:53.354+0000 7f04c043a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:01:53.634 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:53.354+0000 7f04c043a640 -1 osd.4 59 *** Got signal Terminated *** 2024-09-20T19:01:53.634 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:53.354+0000 7f04c043a640 -1 osd.4 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:01:54.085 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:53.644+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:54.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:53.846+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:54.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:53.734+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:54.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:53 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:53.626+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:54.844 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:54 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:54.675+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:54.845 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:54 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:54.833+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:54.845 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:54 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:54.717+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:54.845 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:54 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:54.601+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:55.605 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:55 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:55.596+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:55.882 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:55 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:55.668+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:55.882 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:55 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:55.661+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:55.882 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:55 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:55.661+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:30.779387+0000 front 2024-09-20T19:01:30.779332+0000 (oldest deadline 2024-09-20T19:01:54.879134+0000) 2024-09-20T19:01:56.209 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:55 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:55.874+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:56.893 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:56 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:56.636+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:56.893 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:56 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:56.633+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:56.894 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:56 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:56.619+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:56.894 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:56 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:56.619+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:30.779387+0000 front 2024-09-20T19:01:30.779332+0000 (oldest deadline 2024-09-20T19:01:54.879134+0000) 2024-09-20T19:01:57.222 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:56 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:56.922+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:57.222 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:56 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:56.922+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:01:57.930 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:57.583+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.978481+0000 front 2024-09-20T19:01:21.978388+0000 (oldest deadline 2024-09-20T19:01:46.678347+0000) 2024-09-20T19:01:57.930 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4[32038]: 2024-09-20T19:01:57.583+0000 7f04b8a46640 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:30.779387+0000 front 2024-09-20T19:01:30.779332+0000 (oldest deadline 2024-09-20T19:01:54.879134+0000) 2024-09-20T19:01:57.930 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:57.686+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:57.931 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:57.587+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:57.931 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:57.588+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:01:58.335 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:57.923+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:58.335 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:57 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:57.923+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:01:58.836 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:58 smithi194 podman[54558]: 2024-09-20 19:01:58.795620015 +0000 UTC m=+5.553433174 container died f9660ee9ed812a01ea8be010e4f3f76651f5e9449dbd53ac807316ad5828c1df (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T19:01:58.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:58 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:58.660+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:58.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:58 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:58.660+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:01:58.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:58 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:58.555+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:58.837 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:58 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:58.555+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:01:59.336 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 podman[54558]: 2024-09-20 19:01:59.125031088 +0000 UTC m=+5.882844257 container cleanup f9660ee9ed812a01ea8be010e4f3f76651f5e9449dbd53ac807316ad5828c1df (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T19:01:59.336 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 bash[54558]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4 2024-09-20T19:01:59.336 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:58 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:58.965+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:59.336 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:58 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:58.965+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:01:59.700 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 podman[55165]: 2024-09-20 19:01:59.334516416 +0000 UTC m=+0.527429027 container remove f9660ee9ed812a01ea8be010e4f3f76651f5e9449dbd53ac807316ad5828c1df (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:01:59.700 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:59 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:59.516+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:01:59.700 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:01:59 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:01:59.516+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:01:59.994 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:59 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:59.935+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:01:59.994 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:01:59 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:01:59.935+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:01:59.995 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:59 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:59.692+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:01:59.995 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:01:59 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:01:59.692+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:01:59.995 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 podman[55250]: 2024-09-20 19:01:59.641697528 +0000 UTC m=+0.015889444 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:01:59.995 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 podman[55250]: 2024-09-20 19:01:59.760634326 +0000 UTC m=+0.134826273 container create b48e7c4209206826591517bd313be9ebd809a1b691e16b15a4f416484a7b41b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:02:00.336 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 podman[55250]: 2024-09-20 19:01:59.987697111 +0000 UTC m=+0.361889026 container init b48e7c4209206826591517bd313be9ebd809a1b691e16b15a4f416484a7b41b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T19:02:00.336 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:01:59 smithi194 podman[55250]: 2024-09-20 19:01:59.991074475 +0000 UTC m=+0.365266386 container start b48e7c4209206826591517bd313be9ebd809a1b691e16b15a4f416484a7b41b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:02:00.336 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:02:00 smithi194 podman[55250]: 2024-09-20 19:02:00.079387792 +0000 UTC m=+0.453579700 container attach b48e7c4209206826591517bd313be9ebd809a1b691e16b15a4f416484a7b41b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4-deactivate, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T19:02:00.336 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 20 19:02:00 smithi194 podman[55250]: 2024-09-20 19:02:00.17182639 +0000 UTC m=+0.546018310 container died b48e7c4209206826591517bd313be9ebd809a1b691e16b15a4f416484a7b41b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T19:02:00.806 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:00 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:00.691+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:00.806 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:00 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:00.691+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:00.806 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:00 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:00.526+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:00.807 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:00 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:00.526+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:00.823 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.4.service' 2024-09-20T19:02:00.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:02:00.863 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-20T19:02:00.863 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-20T19:02:00.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.5 2024-09-20T19:02:01.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:00 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:00.888+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:02:01.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:00 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:00.888+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:02:01.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:00 smithi194 systemd[1]: Stopping Ceph osd.5 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:01.575 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:01.177+0000 7fd8c36d7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:02:01.576 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:01.177+0000 7fd8c36d7640 -1 osd.5 59 *** Got signal Terminated *** 2024-09-20T19:02:01.576 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:01.177+0000 7fd8c36d7640 -1 osd.5 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:02:01.835 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:01.661+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:01.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:01.661+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:01.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:01.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:01.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:01.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:02.335 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:01.854+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:02:02.335 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:01 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:01.854+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:02:02.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:02.561+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:02.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:02.561+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:02.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:02.684+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:02.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:02.684+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:03.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:02.828+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:02:03.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:02 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:02.828+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:02:03.835 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:03.659+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:03.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:03.659+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:03.836 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:03.781+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:02:03.836 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:03.781+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:02:03.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:03.533+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:03.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:03 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:03.533+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:04.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:04.541+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:04.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:04.541+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:04.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:04.655+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:04.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:04.655+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:04.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:04.655+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:05.335 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:04.828+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:02:05.336 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:04 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:04.828+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:02:06.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:05.814+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:23.892244+0000 front 2024-09-20T19:01:23.892132+0000 (oldest deadline 2024-09-20T19:01:47.992043+0000) 2024-09-20T19:02:06.086 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5[35465]: 2024-09-20T19:02:05.814+0000 7fd8bfceb640 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.692774+0000 front 2024-09-20T19:01:32.692784+0000 (oldest deadline 2024-09-20T19:01:56.792627+0000) 2024-09-20T19:02:06.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:05.588+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:06.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:05.588+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:06.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:05.687+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:06.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:05.687+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:06.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:05 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:05.687+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:06.826 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:06 smithi194 podman[55350]: 2024-09-20 19:02:06.57211724 +0000 UTC m=+5.500802407 container died 27b9e617a9e625fdfdbc45fa5fb09a22213017e48476df2cc33d4c38082afb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T19:02:06.827 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:06.709+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:06.827 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:06.709+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:06.827 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:06.709+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:06.827 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:06.547+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:06.827 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:06.547+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:06.827 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:06 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:06.547+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:07.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:06 smithi194 podman[55350]: 2024-09-20 19:02:06.968894038 +0000 UTC m=+5.897579204 container cleanup 27b9e617a9e625fdfdbc45fa5fb09a22213017e48476df2cc33d4c38082afb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-20T19:02:07.085 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:06 smithi194 bash[55350]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5 2024-09-20T19:02:07.562 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:07 smithi194 podman[55365]: 2024-09-20 19:02:07.178654505 +0000 UTC m=+0.603563915 container remove 27b9e617a9e625fdfdbc45fa5fb09a22213017e48476df2cc33d4c38082afb64 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-20T19:02:07.835 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:07.694+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:07.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:07.694+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:07.836 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:07.694+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:07.836 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:07 smithi194 podman[55459]: 2024-09-20 19:02:07.487396936 +0000 UTC m=+0.015828722 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:02:07.836 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:07 smithi194 podman[55459]: 2024-09-20 19:02:07.595990592 +0000 UTC m=+0.124422424 container create 807a334714c716bb0ee3bf691d9b38ec60e9cf5b42e53b90f34b8ff07ce8b80d (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:02:07.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:07.554+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:07.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:07.554+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:07.837 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:07 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:07.554+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:08.335 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:07 smithi194 podman[55459]: 2024-09-20 19:02:07.849019058 +0000 UTC m=+0.377450854 container init 807a334714c716bb0ee3bf691d9b38ec60e9cf5b42e53b90f34b8ff07ce8b80d (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d) 2024-09-20T19:02:08.336 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:07 smithi194 podman[55459]: 2024-09-20 19:02:07.8522082 +0000 UTC m=+0.380639991 container start 807a334714c716bb0ee3bf691d9b38ec60e9cf5b42e53b90f34b8ff07ce8b80d (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:02:08.336 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:07 smithi194 podman[55459]: 2024-09-20 19:02:07.948472878 +0000 UTC m=+0.476904678 container attach 807a334714c716bb0ee3bf691d9b38ec60e9cf5b42e53b90f34b8ff07ce8b80d (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-20T19:02:08.336 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 20 19:02:08 smithi194 podman[55459]: 2024-09-20 19:02:08.034377686 +0000 UTC m=+0.562809482 container died 807a334714c716bb0ee3bf691d9b38ec60e9cf5b42e53b90f34b8ff07ce8b80d (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:02:08.632 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:08.583+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:08.632 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:08.583+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:08.632 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:08.583+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:08.663 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.5.service' 2024-09-20T19:02:08.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:02:08.701 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-20T19:02:08.702 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-20T19:02:08.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.6 2024-09-20T19:02:09.036 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:08.680+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:09.036 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:08.680+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:09.036 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:08 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:08.680+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:09.036 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:08 smithi194 systemd[1]: Stopping Ceph osd.6 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:09.036 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:09.028+0000 7f58b3a96640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:02:09.037 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:09.028+0000 7f58b3a96640 -1 osd.6 59 *** Got signal Terminated *** 2024-09-20T19:02:09.037 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:09.028+0000 7f58b3a96640 -1 osd.6 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:02:10.085 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:09.687+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:10.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:09.687+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:10.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:09.687+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:10.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:09.581+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:10.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:09.581+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:10.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:09 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:09.581+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:10.936 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:10.621+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:10.936 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:10.621+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:10.936 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:10.621+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:10.936 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:10.661+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:10.936 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:10.661+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:10.937 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:10 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:10.661+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:11.598+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:11.598+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:11.598+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:11.639+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:11.639+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:11.639+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:11.952 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:11 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:11.639+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:50.535372+0000 front 2024-09-20T19:01:50.535299+0000 (oldest deadline 2024-09-20T19:02:11.635220+0000) 2024-09-20T19:02:12.687 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:12.566+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:12.687 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:12.566+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:12.687 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:12.566+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:13.053 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:12.679+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:13.053 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:12.679+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:13.053 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:12.679+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:13.053 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:12 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:12.679+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:50.535372+0000 front 2024-09-20T19:01:50.535299+0000 (oldest deadline 2024-09-20T19:02:11.635220+0000) 2024-09-20T19:02:14.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:13.674+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.832685+0000 front 2024-09-20T19:01:21.832707+0000 (oldest deadline 2024-09-20T19:01:46.532446+0000) 2024-09-20T19:02:14.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:13.674+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:32.333711+0000 front 2024-09-20T19:01:32.333663+0000 (oldest deadline 2024-09-20T19:01:58.233601+0000) 2024-09-20T19:02:14.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:13.674+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:38.734700+0000 front 2024-09-20T19:01:38.734601+0000 (oldest deadline 2024-09-20T19:02:04.634419+0000) 2024-09-20T19:02:14.086 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6[38908]: 2024-09-20T19:02:13.674+0000 7f58ac0a2640 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:50.535372+0000 front 2024-09-20T19:01:50.535299+0000 (oldest deadline 2024-09-20T19:02:11.635220+0000) 2024-09-20T19:02:14.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:13.594+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:14.087 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:13.594+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:14.087 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:13 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:13.594+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:14.766 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:14 smithi194 podman[55559]: 2024-09-20 19:02:14.459017634 +0000 UTC m=+5.545692858 container died 80cec7e59f6b89f540b51332a345a8c26c1fda441dc68c74fd79713a1d5fe0eb (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T19:02:14.766 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:14 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:14.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:14.766 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:14 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:14.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:14.766 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:14 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:14.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:14.767 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:14 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:14.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:15.085 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:14 smithi194 podman[55559]: 2024-09-20 19:02:14.856255706 +0000 UTC m=+5.942930927 container cleanup 80cec7e59f6b89f540b51332a345a8c26c1fda441dc68c74fd79713a1d5fe0eb (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-20T19:02:15.085 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:14 smithi194 bash[55559]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6 2024-09-20T19:02:15.484 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55797]: 2024-09-20 19:02:15.126279283 +0000 UTC m=+0.663351394 container remove 80cec7e59f6b89f540b51332a345a8c26c1fda441dc68c74fd79713a1d5fe0eb (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:15 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:15.546+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:15 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:15.546+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:15 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:15.546+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:15 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:15.546+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55918]: 2024-09-20 19:02:15.382398674 +0000 UTC m=+0.019445916 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55918]: 2024-09-20 19:02:15.485316773 +0000 UTC m=+0.122363996 container create db4ec2f52c992c77f558251996273d2373067da7f57712c9cf223f7eecba4d62 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55918]: 2024-09-20 19:02:15.695662777 +0000 UTC m=+0.332710013 container init db4ec2f52c992c77f558251996273d2373067da7f57712c9cf223f7eecba4d62 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6-deactivate, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T19:02:15.786 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55918]: 2024-09-20 19:02:15.698977026 +0000 UTC m=+0.336024246 container start db4ec2f52c992c77f558251996273d2373067da7f57712c9cf223f7eecba4d62 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T19:02:15.787 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55918]: 2024-09-20 19:02:15.778993257 +0000 UTC m=+0.416040488 container attach db4ec2f52c992c77f558251996273d2373067da7f57712c9cf223f7eecba4d62 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T19:02:16.085 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 20 19:02:15 smithi194 podman[55918]: 2024-09-20 19:02:15.891365073 +0000 UTC m=+0.528412306 container died db4ec2f52c992c77f558251996273d2373067da7f57712c9cf223f7eecba4d62 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T19:02:16.476 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.6.service' 2024-09-20T19:02:16.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:02:16.520 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-20T19:02:16.520 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-20T19:02:16.520 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.7 2024-09-20T19:02:16.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.550+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:16.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.550+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:16.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.550+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:16.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.550+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:16.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 systemd[1]: Stopping Ceph osd.7 for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:17.327 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.858+0000 7f17c3955640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:02:17.328 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.858+0000 7f17c3955640 -1 osd.7 59 *** Got signal Terminated *** 2024-09-20T19:02:17.328 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:16 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:16.858+0000 7f17c3955640 -1 osd.7 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:02:17.585 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:17 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:17.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:17.585 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:17 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:17.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:17.585 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:17 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:17.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:17.585 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:17 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:17.559+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:18.670 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:18.604+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:18.670 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:18.604+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:18.670 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:18.604+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:18.670 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:18 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:18.604+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:20.085 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:19.602+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:20.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:19.602+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:20.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:19.602+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:20.086 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:19 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:19.602+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:20.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:20.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:20.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:20.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:20.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:20.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:20.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:20.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:20.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:20 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:20.568+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.194:6804 osd.4 since back 2024-09-20T19:01:57.842061+0000 front 2024-09-20T19:01:57.842095+0000 (oldest deadline 2024-09-20T19:02:20.142073+0000) 2024-09-20T19:02:21.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:21 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:21.525+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-09-20T19:01:21.038327+0000 front 2024-09-20T19:01:21.038306+0000 (oldest deadline 2024-09-20T19:01:45.738146+0000) 2024-09-20T19:02:21.835 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:21 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:21.525+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6814 osd.1 since back 2024-09-20T19:01:31.038957+0000 front 2024-09-20T19:01:31.039050+0000 (oldest deadline 2024-09-20T19:01:56.938732+0000) 2024-09-20T19:02:21.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:21 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:21.525+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6822 osd.2 since back 2024-09-20T19:01:41.540275+0000 front 2024-09-20T19:01:41.540258+0000 (oldest deadline 2024-09-20T19:02:06.240059+0000) 2024-09-20T19:02:21.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:21 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:21.525+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.76:6830 osd.3 since back 2024-09-20T19:01:49.741099+0000 front 2024-09-20T19:01:49.741151+0000 (oldest deadline 2024-09-20T19:02:14.341329+0000) 2024-09-20T19:02:21.836 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:21 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7[42394]: 2024-09-20T19:02:21.525+0000 7f17bff69640 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.194:6804 osd.4 since back 2024-09-20T19:01:57.842061+0000 front 2024-09-20T19:01:57.842095+0000 (oldest deadline 2024-09-20T19:02:20.142073+0000) 2024-09-20T19:02:22.635 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:22 smithi194 podman[56097]: 2024-09-20 19:02:22.373561091 +0000 UTC m=+5.617183872 container died e7b6d01d2687ea35c9009b33cb32cd0340d58081d4ea9d8f38d60a37b40d8cab (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T19:02:22.904 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:22 smithi194 podman[56097]: 2024-09-20 19:02:22.810454443 +0000 UTC m=+6.054077223 container cleanup e7b6d01d2687ea35c9009b33cb32cd0340d58081d4ea9d8f38d60a37b40d8cab (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T19:02:22.904 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:22 smithi194 bash[56097]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7 2024-09-20T19:02:23.335 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56496]: 2024-09-20 19:02:23.003348305 +0000 UTC m=+0.622717159 container remove e7b6d01d2687ea35c9009b33cb32cd0340d58081d4ea9d8f38d60a37b40d8cab (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:02:23.335 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:22.897Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:02:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:22.897Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:02:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:22.899Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:02:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:22.899Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:02:23.336 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:22 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:22.900Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.76:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.76:8765: connect: connection refused" 2024-09-20T19:02:23.758 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56580]: 2024-09-20 19:02:23.324198807 +0000 UTC m=+0.016240023 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:02:23.758 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56580]: 2024-09-20 19:02:23.4456292 +0000 UTC m=+0.137670411 container create 02bb61cf566cf071fa0fc976a9dc9fe7e9c1fb8911e62ee6f486db7e99c9dd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T19:02:23.758 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56580]: 2024-09-20 19:02:23.656244419 +0000 UTC m=+0.348285640 container init 02bb61cf566cf071fa0fc976a9dc9fe7e9c1fb8911e62ee6f486db7e99c9dd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS) 2024-09-20T19:02:23.758 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56580]: 2024-09-20 19:02:23.65950147 +0000 UTC m=+0.351542684 container start 02bb61cf566cf071fa0fc976a9dc9fe7e9c1fb8911e62ee6f486db7e99c9dd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-20T19:02:23.758 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56580]: 2024-09-20 19:02:23.75125552 +0000 UTC m=+0.443296753 container attach 02bb61cf566cf071fa0fc976a9dc9fe7e9c1fb8911e62ee6f486db7e99c9dd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-20T19:02:24.085 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 20 19:02:23 smithi194 podman[56580]: 2024-09-20 19:02:23.856034018 +0000 UTC m=+0.548075238 container died 02bb61cf566cf071fa0fc976a9dc9fe7e9c1fb8911e62ee6f486db7e99c9dd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T19:02:24.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@osd.7.service' 2024-09-20T19:02:24.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:02:24.569 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-20T19:02:24.569 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-20T19:02:24.569 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-eefb3510-7780-11ef-baf5-efdc52797490@prometheus.a 2024-09-20T19:02:24.835 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 systemd[1]: Stopping Ceph prometheus.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:25.153 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:02:25.153 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T19:02:25.153 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T19:02:25.153 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T19:02:25.153 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T19:02:25.153 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.879Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.880Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.880Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a[49746]: ts=2024-09-20T19:02:24.880Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T19:02:25.154 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:24 smithi194 podman[56685]: 2024-09-20 19:02:24.889945181 +0000 UTC m=+0.108565639 container died a74adbec870e5c87d60a2d886b8e5246f16e0db6a3311028cf810213dd9f42b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:25.536 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:25 smithi194 podman[56685]: 2024-09-20 19:02:25.28711348 +0000 UTC m=+0.505733938 container cleanup a74adbec870e5c87d60a2d886b8e5246f16e0db6a3311028cf810213dd9f42b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:25.536 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 20 19:02:25 smithi194 bash[56685]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T19:02:25.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eefb3510-7780-11ef-baf5-efdc52797490@prometheus.a.service' 2024-09-20T19:02:25.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:02:25.701 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-20T19:02:25.701 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eefb3510-7780-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T19:02:25.843 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T19:02:27.535 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:27 smithi076 systemd[1]: Stopping Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:27.826 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:27 smithi076 ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a[59947]: ts=2024-09-20T19:02:27.527Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:02:27.826 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:27 smithi076 podman[77053]: 2024-09-20 19:02:27.558523875 +0000 UTC m=+0.141928258 container died 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:28.386 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 podman[77053]: 2024-09-20 19:02:28.103119657 +0000 UTC m=+0.686524069 container cleanup 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:28.386 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 bash[77053]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T19:02:28.639 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 podman[77067]: 2024-09-20 19:02:28.379256714 +0000 UTC m=+0.818523233 container remove 93ee0f5e2d2586a0af8082b17aab0163ea2bd1d962ce4b1441e125ef2e07e765 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:28.639 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 podman[77067]: 2024-09-20 19:02:28.463226772 +0000 UTC m=+0.902493242 volume remove 5bd3e566ee02a8d0c7f40e7a62e9218a055b00d8e1dcbc8e6d2b4e50264caf5f 2024-09-20T19:02:29.030 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:28 smithi076 systemd[1]: Stopping Ceph node-exporter.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:29.030 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T19:02:29.030 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 systemd[1]: Stopped Ceph alertmanager.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T19:02:29.030 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Sep 20 19:02:28 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@alertmanager.a.service: Consumed 1.159s CPU time. 2024-09-20T19:02:29.364 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 podman[77173]: 2024-09-20 19:02:29.078691957 +0000 UTC m=+0.126459477 container died b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:29.773 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 podman[77173]: 2024-09-20 19:02:29.548995038 +0000 UTC m=+0.596762552 container cleanup b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:29.773 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 bash[77173]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a 2024-09-20T19:02:30.029 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 podman[77185]: 2024-09-20 19:02:29.766891309 +0000 UTC m=+0.688415071 container remove b2b4c359219a944c55fc7aa33cc2913554dccc0bfe09aaa73e184a140b21ad92 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T19:02:30.030 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:02:30.030 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-20T19:02:30.030 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 systemd[1]: Stopped Ceph node-exporter.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T19:02:30.030 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Sep 20 19:02:29 smithi076 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.a.service: Consumed 2.350s CPU time. 2024-09-20T19:02:30.606 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eefb3510-7780-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T19:02:30.757 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T19:02:32.335 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:32 smithi194 systemd[1]: Stopping Ceph node-exporter.b for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:32.764 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:32 smithi194 podman[57129]: 2024-09-20 19:02:32.427523415 +0000 UTC m=+0.102518846 container died ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T19:02:32.765 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:32 smithi194 podman[57129]: 2024-09-20 19:02:32.758149808 +0000 UTC m=+0.433145236 container cleanup ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T19:02:33.085 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:32 smithi194 bash[57129]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b 2024-09-20T19:02:33.085 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:32 smithi194 podman[57142]: 2024-09-20 19:02:32.934237702 +0000 UTC m=+0.506461829 container remove ab73fcdb34139cf6f0234aeaa60f7641e83f08f3faa676ac7707ca87d57680cf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T19:02:33.085 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:32 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:02:33.586 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:33 smithi194 systemd[1]: Stopping Ceph grafana.a for eefb3510-7780-11ef-baf5-efdc52797490... 2024-09-20T19:02:33.587 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:33 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-20T19:02:33.587 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:33 smithi194 systemd[1]: Stopped Ceph node-exporter.b for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T19:02:33.587 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 20 19:02:33 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@node-exporter.b.service: Consumed 2.279s CPU time. 2024-09-20T19:02:33.934 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:33 smithi194 podman[57249]: 2024-09-20 19:02:33.61433007 +0000 UTC m=+0.123856645 container died 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, architecture=x86_64, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897) 2024-09-20T19:02:33.935 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:33 smithi194 podman[57249]: 2024-09-20 19:02:33.928299513 +0000 UTC m=+0.437826134 container cleanup 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a, maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, release=239.1651231664, version=8.5) 2024-09-20T19:02:34.325 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:33 smithi194 bash[57249]: ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a 2024-09-20T19:02:34.326 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:34 smithi194 podman[57261]: 2024-09-20 19:02:34.09593548 +0000 UTC m=+0.483608992 container remove 903b8640ec8195475a7051e2df66bc03705bf7e691b12778567cbb3d27a739a9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-eefb3510-7780-11ef-baf5-efdc52797490-grafana-a, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, architecture=x86_64, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, release=239.1651231664, version=8.5, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git) 2024-09-20T19:02:34.326 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:34 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:02:34.358 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:34 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-20T19:02:34.358 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:34 smithi194 systemd[1]: Stopped Ceph grafana.a for eefb3510-7780-11ef-baf5-efdc52797490. 2024-09-20T19:02:34.358 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 20 19:02:34 smithi194 systemd[1]: ceph-eefb3510-7780-11ef-baf5-efdc52797490@grafana.a.service: Consumed 6.218s CPU time. 2024-09-20T19:02:34.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:02:35.007 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:02:35.034 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-20T19:02:35.037 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726/remote/smithi076/crash 2024-09-20T19:02:35.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/crash -- . 2024-09-20T19:02:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T19:02:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2024-09-20T19:02:35.091 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726/remote/smithi194/crash 2024-09-20T19:02:35.092 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/crash -- . 2024-09-20T19:02:35.125 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/eefb3510-7780-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T19:02:35.125 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-09-20T19:02:35.126 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-20T19:02:35.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-20T19:02:35.158 INFO:tasks.cephadm:Compressing logs... 2024-09-20T19:02:35.159 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T19:02:35.201 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T19:02:35.224 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-20T19:02:35.224 INFO:teuthology.orchestra.run.smithi076.stderr:: No such file or directory 2024-09-20T19:02:35.224 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-20T19:02:35.224 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T19:02:35.224 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.0.log: /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T19:02:35.224 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-20T19:02:35.225 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-volume.log: /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T19:02:35.225 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-20T19:02:35.226 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-20T19:02:35.227 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-20T19:02:35.227 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-09-20T19:02:35.227 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T19:02:35.227 INFO:teuthology.orchestra.run.smithi194.stderr:gzip/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.7.log: -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-20T19:02:35.227 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T19:02:35.228 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.a.log: 89.1% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T19:02:35.228 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-20T19:02:35.228 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log 2024-09-20T19:02:35.228 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.2.log: /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mgr.y.log: 94.1% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T19:02:35.229 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-volume.log: /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T19:02:35.229 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-20T19:02:35.229 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-20T19:02:35.229 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.6.log: 79.7% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T19:02:35.229 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log 2024-09-20T19:02:35.230 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-20T19:02:35.230 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log: /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mgr.x.log: 90.5% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T19:02:35.230 INFO:teuthology.orchestra.run.smithi194.stderr: 87.8% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T19:02:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-20T19:02:35.235 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T19:02:35.236 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-20T19:02:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-20T19:02:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log: 93.0% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T19:02:35.241 INFO:teuthology.orchestra.run.smithi194.stderr: 82.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T19:02:35.251 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T19:02:35.256 INFO:teuthology.orchestra.run.smithi194.stderr: 91.4% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T19:02:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.3.log: 91.3% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T19:02:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cephadm.log: 89.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T19:02:35.297 INFO:teuthology.orchestra.run.smithi194.stderr: 92.0% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-20T19:02:35.310 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-20T19:02:35.326 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-20T19:02:35.331 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-20T19:02:35.335 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-20T19:02:35.345 INFO:teuthology.orchestra.run.smithi076.stderr: 92.2% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-20T19:02:35.350 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-20T19:02:35.363 INFO:teuthology.orchestra.run.smithi076.stderr: 92.9% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-20T19:02:35.372 INFO:teuthology.orchestra.run.smithi076.stderr: 88.0% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-20T19:02:35.384 INFO:teuthology.orchestra.run.smithi194.stderr: 92.5% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-20T19:02:35.386 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-09-20T19:02:35.386 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m0.171s 2024-09-20T19:02:35.386 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m0.548s 2024-09-20T19:02:35.386 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.039s 2024-09-20T19:02:35.410 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-20T19:02:35.658 INFO:teuthology.orchestra.run.smithi076.stderr: 90.8% -- replaced with /var/log/ceph/eefb3510-7780-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-20T19:02:35.660 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-20T19:02:35.660 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m0.447s 2024-09-20T19:02:35.660 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m1.268s 2024-09-20T19:02:35.660 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.050s 2024-09-20T19:02:35.660 INFO:tasks.cephadm:Archiving logs... 2024-09-20T19:02:35.660 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726/remote/smithi076/log 2024-09-20T19:02:35.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T19:02:35.900 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726/remote/smithi194/log 2024-09-20T19:02:35.901 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T19:02:36.007 INFO:tasks.cephadm:Removing cluster... 2024-09-20T19:02:36.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eefb3510-7780-11ef-baf5-efdc52797490 --force 2024-09-20T19:02:36.147 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T19:02:36.276 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eefb3510-7780-11ef-baf5-efdc52797490 --force 2024-09-20T19:02:36.415 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: eefb3510-7780-11ef-baf5-efdc52797490 2024-09-20T19:02:36.541 INFO:tasks.cephadm:Removing cephadm ... 2024-09-20T19:02:36.541 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T19:02:36.557 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T19:02:36.573 INFO:tasks.cephadm:Teardown complete 2024-09-20T19:02:36.573 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/cephadm.py", line 2371, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/ceph.py", line 1537, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/ceph_manager.py", line 3303, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-09-20T19:02:36.749 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7ee03ae19d504213a01d06ca5a430fa7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/cephadm.py", line 2371, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/ceph.py", line 1537, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks/ceph_manager.py", line 3303, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-09-20T19:02:36.752 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-20T19:02:36.762 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-20T19:02:36.801 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-20T19:02:36.802 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:02:36.814 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:02:36.819 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-09-20T19:02:36.824 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T19:02:36.824 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-09-20T19:02:36.824 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +126us[ +126us] +/- 39ms 2024-09-20T19:02:36.824 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 +1288us[+1288us] +/- 60ms 2024-09-20T19:02:36.824 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -1644us[-1644us] +/- 62ms 2024-09-20T19:02:36.824 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:02:36.829 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-09-20T19:02:36.834 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T19:02:36.834 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-09-20T19:02:36.834 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +151us[ +145us] +/- 39ms 2024-09-20T19:02:36.834 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 +1311us[+1306us] +/- 60ms 2024-09-20T19:02:36.834 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -1640us[-1646us] +/- 62ms 2024-09-20T19:02:36.834 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:02:36.835 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-20T19:02:36.851 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-20T19:02:36.852 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-20T19:02:36.902 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T19:02:36.973 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T19:02:37.044 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T19:02:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:02:37.072 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T19:02:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:02:37.100 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-20T19:02:37.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-20T19:02:37.136 INFO:teuthology.task.internal:Duration was 1069.993279 seconds 2024-09-20T19:02:37.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-20T19:02:37.190 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-20T19:02:37.191 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T19:02:37.193 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T19:02:37.231 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T19:02:37.235 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T19:02:37.563 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-20T19:02:37.563 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-09-20T19:02:37.563 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T19:02:37.583 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-09-20T19:02:37.583 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T19:02:37.611 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-20T19:02:37.611 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T19:02:37.625 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T19:02:37.676 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-20T19:02:37.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T19:02:37.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T19:02:38.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-20T19:02:38.036 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-20T19:02:38.036 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T19:02:38.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T19:02:38.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-20T19:02:38.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:02:38.105 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:02:38.125 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-09-20T19:02:38.155 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-09-20T19:02:38.179 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:02:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:02:38.204 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:02:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:02:38.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-20T19:02:38.248 INFO:teuthology.task.internal:Transferring archived files... 2024-09-20T19:02:38.249 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726/remote/smithi076 2024-09-20T19:02:38.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T19:02:38.290 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912726/remote/smithi194 2024-09-20T19:02:38.290 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T19:02:38.326 INFO:teuthology.task.internal:Removing archive directory... 2024-09-20T19:02:38.327 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T19:02:38.329 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T19:02:38.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-20T19:02:38.386 INFO:teuthology.task.internal:Not uploading archives. 2024-09-20T19:02:38.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-20T19:02:38.419 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-20T19:02:38.419 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T19:02:38.421 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T19:02:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 19:02 /home/ubuntu/cephtest 2024-09-20T19:02:38.435 INFO:teuthology.orchestra.run.smithi194.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 19:02 /home/ubuntu/cephtest 2024-09-20T19:02:38.436 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-20T19:02:38.468 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-20T19:02:38.572 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} duration: 1069.9932792186737 failure_reason: timeout expired in wait_until_healthy owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7ee03ae19d504213a01d06ca5a430fa7 status: fail success: false 2024-09-20T19:02:38.572 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T19:02:38.649 INFO:teuthology.run:FAIL